From 4f68df74762360552f8a0354325514c55c6e6e8c Mon Sep 17 00:00:00 2001 From: Teodor Calin Date: Thu, 28 May 2026 12:50:33 -0700 Subject: [PATCH 1/4] ci: document pending org-move secret migration in release.yml header MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop a TODO at the top of release.yml enumerating which secrets need to be re-created on `pilot-protocol/` before the repo transfer, since GitHub secrets do not survive a repo transfer. Currently the only expected secret is GITHUB_TOKEN (auto-issued). HOMEBREW_TAP_TOKEN was removed in #122 when update-homebrew.yml was dropped, and NPM_TOKEN / PYPI_TOKEN / COSIGN_KEY are placeholders for the auto-publish (PILOT-203) and binary-signing (PILOT-114) work that hasn't landed yet. This is documentation only — no behavior change. The comment block is load-bearing for the org migration; deleting it before the new org has its secrets configured will silently break the next release. --- .github/workflows/release.yml | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index deabf21e..95579c64 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,3 +1,36 @@ +# ---------------------------------------------------------------------------- +# TODO(org-move): re-create secrets on the destination org before this repo +# is transferred to `pilot-protocol/`. None of the secrets below survive a +# repo transfer — they live on the SOURCE org (TeoSlayer) and need to be +# created fresh on `pilot-protocol` so this workflow keeps working. +# +# Secrets currently expected by `release.yml` and other workflows in this +# repo (`ci.yml`, `nightly.yml`, `architecture.yml`, `codeql.yml`): +# +# GITHUB_TOKEN - auto-issued, no action needed +# +# Secrets that need to be added BACK once the matching workflows are +# restored (we removed `update-homebrew.yml` in PR #122 to drop the +# cross-org PAT; auto-publish workflows for npm/PyPI never landed): +# +# HOMEBREW_TAP_TOKEN - PAT or GitHub App token with `contents:write` +# on `pilot-protocol/homebrew-pilot`. +# Prefer a GitHub App over a PAT — see +# `actions/create-github-app-token@v1`. +# NPM_TOKEN - if PILOT-203 lands sdk-node auto-publish. +# PYPI_TOKEN - if PILOT-203 lands sdk-python auto-publish. +# COSIGN_KEY / COSIGN_PASS - if PILOT-114 lands updater binary signing. +# +# When the migration happens, mirror the secrets via: +# gh secret set HOMEBREW_TAP_TOKEN --repo pilot-protocol/ --body +# (Reading the value from the old org first; GitHub never exposes secrets, +# so the original cleartext source is required.) +# +# Track the migration in the org-move runbook; do not delete this comment +# until every workflow file that previously referenced a secret has either +# (a) been re-wired against the new secret, or (b) been confirmed retired. +# ---------------------------------------------------------------------------- + name: Release on: From c755a07bfcac0014e1191d53000e7f3f88ae0b25 Mon Sep 17 00:00:00 2001 From: Teodor Calin Date: Thu, 28 May 2026 15:24:29 -0700 Subject: [PATCH 2/4] release: publish manifest + shockwave fan-out on tag MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two new jobs run after the existing release job: publish-manifest builds public/.well-known/latest.json from the tag's checksums.txt, then repository_dispatches it to pilot-protocol/website, which commits the JSON to main and triggers the Cloudflare deploy. The single canonical manifest at pilotprotocol.network/.well-known/latest.json is consumed by install.sh, the Homebrew formula bump workflow, and the SDK release helpers — one shockwave per release. shockwave fans out repository_dispatch(event_type=upstream-release) to homebrew-pilot, sdk-node, sdk-python, and sdk-swift so each consumer can run its own bump workflow. Per-target dispatch is soft-fail with a summary so a missing token on one repo does not block the others. Both jobs require a new SHOCKWAVE_DISPATCH_TOKEN secret with repository_dispatch scope on each downstream repo (prefer a GitHub App token over a PAT). When the secret is absent the steps emit a clear ::warning:: and exit 0 so existing release flow is not broken. --- .github/workflows/release.yml | 211 ++++++++++++++++++++++++++++++---- 1 file changed, 189 insertions(+), 22 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 95579c64..d0c991d7 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,34 +1,38 @@ # ---------------------------------------------------------------------------- -# TODO(org-move): re-create secrets on the destination org before this repo -# is transferred to `pilot-protocol/`. None of the secrets below survive a -# repo transfer — they live on the SOURCE org (TeoSlayer) and need to be -# created fresh on `pilot-protocol` so this workflow keeps working. +# Releases continue from this repo as normal — both before and after the +# planned move to `pilot-protocol/`. This comment is just a reminder for +# migration day; it does NOT gate any release. # -# Secrets currently expected by `release.yml` and other workflows in this -# repo (`ci.yml`, `nightly.yml`, `architecture.yml`, `codeql.yml`): +# Why the reminder: GitHub repo transfers do not carry secrets across orgs. +# The current setup only needs GITHUB_TOKEN (auto-issued), so a transfer +# today would not break anything. But future work will re-introduce +# secrets, and they must be re-created on the destination org BEFORE the +# transfer flips DNS — otherwise the first release after the move silently +# falls back to no-op or fails. # -# GITHUB_TOKEN - auto-issued, no action needed +# Pending (not blocking releases now — will block IF the linked work +# lands before the org migration): # -# Secrets that need to be added BACK once the matching workflows are -# restored (we removed `update-homebrew.yml` in PR #122 to drop the -# cross-org PAT; auto-publish workflows for npm/PyPI never landed): -# -# HOMEBREW_TAP_TOKEN - PAT or GitHub App token with `contents:write` -# on `pilot-protocol/homebrew-pilot`. -# Prefer a GitHub App over a PAT — see -# `actions/create-github-app-token@v1`. +# HOMEBREW_TAP_TOKEN - re-introduced when Homebrew auto-publish +# returns. Prefer a GitHub App over a PAT +# via `actions/create-github-app-token@v1`. # NPM_TOKEN - if PILOT-203 lands sdk-node auto-publish. # PYPI_TOKEN - if PILOT-203 lands sdk-python auto-publish. # COSIGN_KEY / COSIGN_PASS - if PILOT-114 lands updater binary signing. # -# When the migration happens, mirror the secrets via: -# gh secret set HOMEBREW_TAP_TOKEN --repo pilot-protocol/ --body -# (Reading the value from the old org first; GitHub never exposes secrets, -# so the original cleartext source is required.) +# Migration steps (when the day comes): +# 1. List secrets on the source org with `gh secret list --repo `. +# 2. For each non-auto-issued secret, recreate it on the destination +# using the original cleartext value (GitHub never reveals existing +# secret values). +# 3. Transfer the repo via Settings → "Transfer ownership" or +# `gh api repos//transfer -f new_owner=`. +# 4. Re-verify a release tag triggers this workflow successfully. # -# Track the migration in the org-move runbook; do not delete this comment -# until every workflow file that previously referenced a secret has either -# (a) been re-wired against the new secret, or (b) been confirmed retired. +# Track the migration in the org-move runbook. Do NOT delete this comment +# until either: (a) the migration has completed and every reintroduced +# secret is wired against the destination org, or (b) auto-publish and +# binary signing have been formally retired. # ---------------------------------------------------------------------------- name: Release @@ -273,3 +277,166 @@ jobs: generate_release_notes: true draft: false prerelease: ${{ contains(github.ref_name, '-rc') || contains(github.ref_name, '-beta') }} + + # ---------------------------------------------------------------------------- + # publish-manifest + # + # Regenerates `pilotprotocol.network/.well-known/latest.json` from the tag we + # just shipped, then hands it off to `pilot-protocol/website` via + # `repository_dispatch`. The website side commits the JSON to main, which + # triggers the Cloudflare Pages deploy. + # + # The manifest is the single source of truth that every install surface + # (install.sh, Homebrew formula, SDK release helpers) reads to decide which + # version is current. Failing here does NOT roll the release back — the + # GitHub release is already live — but it does mean install.sh will keep + # serving the old tag until the manifest is republished. The step is best- + # effort and prints a clear hint when the dispatch token is missing. + # ---------------------------------------------------------------------------- + publish-manifest: + name: Publish version manifest + needs: release + runs-on: ubuntu-latest + steps: + - name: Build manifest JSON from this release + id: build + env: + TAG: ${{ github.ref_name }} + IS_PRE: ${{ contains(github.ref_name, '-rc') || contains(github.ref_name, '-beta') }} + run: | + # Pull the just-published checksums.txt directly from the GitHub + # release (it landed there in the previous job). + curl -fsSL -o checksums.txt \ + "https://github.com/${GITHUB_REPOSITORY}/releases/download/${TAG}/checksums.txt" + + sha_for() { + grep " $1\$" checksums.txt | awk '{print $1}' + } + + DARWIN_AMD64=$(sha_for "pilot-darwin-amd64.tar.gz") + DARWIN_ARM64=$(sha_for "pilot-darwin-arm64.tar.gz") + LINUX_AMD64=$(sha_for "pilot-linux-amd64.tar.gz") + LINUX_ARM64=$(sha_for "pilot-linux-arm64.tar.gz") + + # Refuse to publish a manifest with missing checksums — install.sh + # would silently skip verification. + for v in "$DARWIN_AMD64" "$DARWIN_ARM64" "$LINUX_AMD64" "$LINUX_ARM64"; do + if [ -z "$v" ]; then + echo "error: checksums.txt missing one or more platform entries" + cat checksums.txt + exit 1 + fi + done + + # When the new tag is a prerelease, leave latest_stable alone and + # bump only latest_prerelease + channels.edge. The website receiver + # merges into the existing manifest. + if [ "$IS_PRE" = "true" ]; then + STABLE="" ; EDGE="$TAG" + else + STABLE="$TAG" ; EDGE="$TAG" + fi + + UPDATED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ) + OWNER_REPO="${GITHUB_REPOSITORY}" + + cat > manifest.json <> "$GITHUB_OUTPUT" + + - name: Dispatch to website + env: + # SHOCKWAVE_DISPATCH_TOKEN must have `repository_dispatch` scope on + # pilot-protocol/website. Prefer a GitHub App token over a PAT — + # see actions/create-github-app-token@v1. + SHOCKWAVE_TOKEN: ${{ secrets.SHOCKWAVE_DISPATCH_TOKEN }} + run: | + if [ -z "$SHOCKWAVE_TOKEN" ]; then + echo "::warning::SHOCKWAVE_DISPATCH_TOKEN secret unset — skipping manifest dispatch." + echo "Action required: set the secret and re-run this workflow to publish ${{ github.ref_name }}." + exit 0 + fi + HTTP_CODE=$(curl -sS -o /tmp/resp -w '%{http_code}' \ + -X POST \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${SHOCKWAVE_TOKEN}" \ + "https://api.github.com/repos/pilot-protocol/website/dispatches" \ + -d '${{ steps.build.outputs.payload }}') + if [ "$HTTP_CODE" != "204" ]; then + echo "dispatch failed (HTTP $HTTP_CODE):" + cat /tmp/resp + exit 1 + fi + echo "manifest dispatch accepted (HTTP 204)" + + # ---------------------------------------------------------------------------- + # shockwave + # + # Notify every package that derives from web4 (Homebrew formula + SDKs) that + # a new release exists. Each downstream repo runs its own bump workflow on + # receiving `repository_dispatch` event_type=upstream-release. + # + # Receivers (each must have a workflow listening for `upstream-release`): + # - pilot-protocol/homebrew-pilot → bump Formula/pilot.rb + # - pilot-protocol/sdk-node → bump pkg version + npm publish + # - pilot-protocol/sdk-python → bump pyproject + PyPI publish + # - pilot-protocol/sdk-swift → bump Package.swift binaryTarget + # + # Soft-fail per receiver: a missing token or a 404 on one repo MUST NOT block + # the others. The job summary at the end lists which targets succeeded so a + # missed dispatch is visible without grepping logs. + # ---------------------------------------------------------------------------- + shockwave: + name: Shockwave fan-out + needs: release + runs-on: ubuntu-latest + steps: + - name: Dispatch to downstream consumers + env: + SHOCKWAVE_TOKEN: ${{ secrets.SHOCKWAVE_DISPATCH_TOKEN }} + TAG: ${{ github.ref_name }} + IS_PRE: ${{ contains(github.ref_name, '-rc') || contains(github.ref_name, '-beta') }} + run: | + if [ -z "$SHOCKWAVE_TOKEN" ]; then + echo "::warning::SHOCKWAVE_DISPATCH_TOKEN secret unset — skipping fan-out." + exit 0 + fi + summary="" + for repo in homebrew-pilot sdk-node sdk-python sdk-swift; do + payload=$(jq -nc --arg tag "$TAG" --argjson pre "$IS_PRE" \ + '{event_type:"upstream-release", client_payload:{tag:$tag, is_prerelease:$pre}}') + HTTP_CODE=$(curl -sS -o /tmp/resp -w '%{http_code}' \ + -X POST \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${SHOCKWAVE_TOKEN}" \ + "https://api.github.com/repos/pilot-protocol/${repo}/dispatches" \ + -d "$payload") + if [ "$HTTP_CODE" = "204" ]; then + summary="${summary} ✓ ${repo}\n" + else + summary="${summary} ✗ ${repo} (HTTP ${HTTP_CODE})\n" + echo "::warning::shockwave dispatch failed for ${repo}: HTTP ${HTTP_CODE}" + cat /tmp/resp + fi + done + printf "Shockwave fan-out summary:\n${summary}" >> "$GITHUB_STEP_SUMMARY" From eaebe1f0cc3e379521b132303fb3adc6d59ed837 Mon Sep 17 00:00:00 2001 From: Teodor Calin Date: Thu, 28 May 2026 15:37:08 -0700 Subject: [PATCH 3/4] ci(orchestrator): central deps.json + reverse-closure fan-out + receiver templates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a central release orchestrator that handles bidirectional dependency propagation: when any package in the constellation ships a release, the set of downstream nodes that need a bump is computed automatically and each gets a dispatch in topological order. Files: .github/deps.json Canonical dependency graph. 25 nodes: hub (web4), 15 Go siblings, 1 Go app, 1 FFI fan-in (libpilot), 3 SDKs, 1 brew tap, 1 surface (website), 3 freeloaders (cosift, pilot-ca, wallet). Each node declares its depends_on edges and a bump_policy (stable_only, auto_commit_main, open_pr_instead_of_main). This is the single source of truth — changes to the graph are PR-reviewed diffs here. .github/workflows/orchestrator.yml Receives package-released dispatches from any node. Computes the reverse-transitive closure of the released package (BFS by depth in deps.json), filters per-receiver stable_only when the upstream is a prerelease, then fan-outs bump-upstream dispatches to each affected repo. Supports workflow_dispatch with a dry_run flag for rehearsal. .github/workflows/_template-emit-release.yml Reusable workflow that any sibling adopts: at the end of its own release.yml it calls this with the node name, and it dispatches package-released to the orchestrator. One token per sibling (ORCHESTRATOR_DISPATCH_TOKEN) covers the hop. .github/workflows/_template-bump-upstream.yml Generic receiver template for Go-sibling repos: receives bump-upstream, runs go mod edit -require=@ + go mod tidy + go build sanity check, then either commits to main or opens a PR based on BUMP_MODE. SDK repos will need their own package-manager-specific variants. Closure smoke-tests pass: web4 → 22 targets (every depending node, ordered by depth) policy → 4 targets (libpilot + 3 SDKs) handshake → 4 targets (libpilot + 3 SDKs) libpilot → 3 targets (3 SDKs) sdk-node → 0 targets (leaf) cosift → 0 targets (freeloader) This SUPERSEDES the direct shockwave fan-out in PR #151's release.yml once orchestrator adoption rolls out — the four hardcoded repos there become entries in deps.json instead. Keeping #151's direct fan-out for now as the bootstrap path (orchestrator needs SHOCKWAVE_DISPATCH_TOKEN set before it can do anything). --- .github/deps.json | 174 +++++++++++++++ .github/workflows/_template-bump-upstream.yml | 139 ++++++++++++ .github/workflows/_template-emit-release.yml | 85 ++++++++ .github/workflows/orchestrator.yml | 204 ++++++++++++++++++ 4 files changed, 602 insertions(+) create mode 100644 .github/deps.json create mode 100644 .github/workflows/_template-bump-upstream.yml create mode 100644 .github/workflows/_template-emit-release.yml create mode 100644 .github/workflows/orchestrator.yml diff --git a/.github/deps.json b/.github/deps.json new file mode 100644 index 00000000..12de2df9 --- /dev/null +++ b/.github/deps.json @@ -0,0 +1,174 @@ +{ + "$schema": "https://pilotprotocol.network/.well-known/deps.schema.json", + "version": 1, + "description": "Canonical dependency graph of every Pilot Protocol repo. Read by .github/workflows/orchestrator.yml to compute which downstream nodes need a bump when any package ships a release. This is the SINGLE source of truth — keep it in sync whenever a repo gains or drops an upstream dependency.", + "nodes": { + "web4": { + "repo": "TeoSlayer/pilotprotocol", + "type": "hub", + "go_module": "github.com/TeoSlayer/pilotprotocol", + "depends_on": [] + }, + + "beacon": { + "repo": "pilot-protocol/beacon", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/beacon", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "dataexchange": { + "repo": "pilot-protocol/dataexchange", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/dataexchange", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "eventstream": { + "repo": "pilot-protocol/eventstream", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/eventstream", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "examples": { + "repo": "pilot-protocol/examples", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/examples", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "gateway": { + "repo": "pilot-protocol/gateway", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/gateway", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "handshake": { + "repo": "pilot-protocol/handshake", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/handshake", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "nameserver": { + "repo": "pilot-protocol/nameserver", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/nameserver", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "policy": { + "repo": "pilot-protocol/policy", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/policy", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "rendezvous": { + "repo": "pilot-protocol/rendezvous", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/rendezvous", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "runtime": { + "repo": "pilot-protocol/runtime", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/runtime", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "skillinject": { + "repo": "pilot-protocol/skillinject", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/skillinject", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "trustedagents": { + "repo": "pilot-protocol/trustedagents", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/trustedagents", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "updater": { + "repo": "pilot-protocol/updater", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/updater", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "webhook": { + "repo": "pilot-protocol/webhook", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol/webhook", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "app-store": { + "repo": "pilot-protocol/app-store", + "type": "go-app", + "go_module": "github.com/TeoSlayer/pilotprotocol/app-store/integration", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + + "libpilot": { + "repo": "pilot-protocol/libpilot", + "type": "ffi-fan-in", + "go_module": "github.com/TeoSlayer/pilotprotocol/libpilot", + "depends_on": ["web4", "handshake", "policy", "runtime", "skillinject"], + "bump_policy": { + "stable_only": true, + "open_pr_instead_of_main": true, + "_note": "FFI fan-in feeds all 3 SDKs at build time — a bad bump cascades to every SDK. Use PR + CI gate." + } + }, + + "sdk-node": { + "repo": "pilot-protocol/sdk-node", + "type": "sdk", + "package_manager": "npm", + "depends_on": ["libpilot"], + "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true } + }, + "sdk-python": { + "repo": "pilot-protocol/sdk-python", + "type": "sdk", + "package_manager": "pypi", + "depends_on": ["libpilot"], + "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true } + }, + "sdk-swift": { + "repo": "pilot-protocol/sdk-swift", + "type": "sdk", + "package_manager": "swiftpm", + "depends_on": ["libpilot"], + "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true } + }, + + "homebrew-pilot": { + "repo": "TeoSlayer/homebrew-pilot", + "type": "package-tap", + "depends_on": ["web4"], + "bump_policy": { "stable_only": true, "auto_commit_main": true } + }, + "website": { + "repo": "pilot-protocol/website", + "type": "surface", + "depends_on": ["web4"], + "bump_policy": { + "stable_only": false, + "auto_commit_main": true, + "_note": "Edge channel also publishes; manifest preserves latest_stable when upstream is prerelease." + } + }, + + "cosift": { "repo": "pilot-protocol/cosift", "type": "freeloader", "depends_on": [] }, + "pilot-ca": { "repo": "pilot-protocol/pilot-ca", "type": "freeloader", "depends_on": [] }, + "wallet": { "repo": "pilot-protocol/wallet", "type": "indirect", "depends_on": ["app-store"] } + } +} diff --git a/.github/workflows/_template-bump-upstream.yml b/.github/workflows/_template-bump-upstream.yml new file mode 100644 index 00000000..c98a0818 --- /dev/null +++ b/.github/workflows/_template-bump-upstream.yml @@ -0,0 +1,139 @@ +name: bump-upstream + +# TEMPLATE — copy this file into a Go-sibling repo's `.github/workflows/` +# and rename to `bump-upstream.yml`. It receives a `bump-upstream` dispatch +# from web4's orchestrator and rewrites the local `go.mod` to require the +# new version of the upstream that just shipped. +# +# Behavior: +# - If client_payload.is_prerelease == true and this repo's policy is +# stable_only (the orchestrator already filtered, but we re-check), skip. +# - `go mod edit -require=@` +# - `go mod tidy` +# - Commit to main OR open a PR depending on BUMP_MODE below. +# +# Tunables (set as repo variables or hardcode): +# BUMP_MODE auto-commit | pr (default: auto-commit) +# UPSTREAM_MODULE_PATH the import path of the upstream — e.g. +# "github.com/TeoSlayer/pilotprotocol". Each +# sibling has the same hub path; FFI nodes like +# libpilot may receive multiple upstreams (the +# orchestrator sends one dispatch per upstream). +# +# This template targets Go modules. SDK repos (sdk-node, sdk-python, +# sdk-swift) need their own variants — see the per-SDK bump templates. + +on: + repository_dispatch: + types: [bump-upstream] + workflow_dispatch: + inputs: + upstream: + description: 'Upstream node name (informational)' + required: true + version: + description: 'Upstream version (e.g. v1.10.6)' + required: true + +permissions: + contents: write + pull-requests: write + +env: + BUMP_MODE: auto-commit + UPSTREAM_MODULE_PATH: github.com/TeoSlayer/pilotprotocol + +jobs: + bump: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-go@v5 + with: + go-version-file: go.mod + + - name: Resolve incoming dispatch + id: input + env: + DISPATCH_UP: ${{ github.event.client_payload.upstream }} + DISPATCH_VER: ${{ github.event.client_payload.version }} + DISPATCH_PRE: ${{ github.event.client_payload.is_prerelease }} + MANUAL_UP: ${{ inputs.upstream }} + MANUAL_VER: ${{ inputs.version }} + run: | + if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then + UP="$MANUAL_UP"; VER="$MANUAL_VER"; PRE="false" + else + UP="$DISPATCH_UP"; VER="$DISPATCH_VER"; PRE="$DISPATCH_PRE" + fi + # Re-check the prerelease gate — defense in depth in case a + # mis-configured orchestrator forgets to filter. + if [ "$PRE" = "true" ]; then + echo "::notice::upstream ${UP} ${VER} is a prerelease; skipping (stable_only policy)" + echo "skip=1" >> "$GITHUB_OUTPUT" + exit 0 + fi + echo "upstream=$UP" >> "$GITHUB_OUTPUT" + echo "version=$VER" >> "$GITHUB_OUTPUT" + echo "skip=0" >> "$GITHUB_OUTPUT" + + - name: Bump go.mod + if: steps.input.outputs.skip != '1' + env: + VER: ${{ steps.input.outputs.version }} + run: | + # The upstream module path is the hub module — siblings require + # it directly. Sub-package nodes (handshake, policy, ...) are + # currently part of the same Go module, so the require line + # always points at the hub path. If a sibling later becomes its + # own top-level Go module, set UPSTREAM_MODULE_PATH per-repo. + go mod edit -require="${UPSTREAM_MODULE_PATH}@${VER}" + go mod tidy + echo "=== updated go.mod ===" + grep "${UPSTREAM_MODULE_PATH}" go.mod || true + + - name: Verify build still works + if: steps.input.outputs.skip != '1' + run: go build ./... || (echo "::error::build broke after bump"; exit 1) + + - name: Commit (auto-commit mode) + if: steps.input.outputs.skip != '1' && env.BUMP_MODE == 'auto-commit' + env: + UP: ${{ steps.input.outputs.upstream }} + VER: ${{ steps.input.outputs.version }} + run: | + if git diff --quiet go.mod go.sum; then + echo "go.mod unchanged — nothing to commit." + exit 0 + fi + git config user.name "pilot-release-bot" + git config user.email "release-bot@pilotprotocol.network" + git add go.mod go.sum + git commit -m "deps: bump ${UP} to ${VER}" \ + -m "Triggered by orchestrator dispatch from upstream release." + git push origin HEAD:main + + - name: Open PR (pr mode) + if: steps.input.outputs.skip != '1' && env.BUMP_MODE == 'pr' + env: + GH_TOKEN: ${{ github.token }} + UP: ${{ steps.input.outputs.upstream }} + VER: ${{ steps.input.outputs.version }} + run: | + if git diff --quiet go.mod go.sum; then + echo "go.mod unchanged — nothing to commit." + exit 0 + fi + BRANCH="bump-${UP}-${VER}" + git config user.name "pilot-release-bot" + git config user.email "release-bot@pilotprotocol.network" + git checkout -b "$BRANCH" + git add go.mod go.sum + git commit -m "deps: bump ${UP} to ${VER}" + git push origin "$BRANCH" + gh pr create \ + --title "deps: bump ${UP} to ${VER}" \ + --body "Orchestrator dispatch from upstream release ${UP} ${VER}." \ + --base main \ + --head "$BRANCH" diff --git a/.github/workflows/_template-emit-release.yml b/.github/workflows/_template-emit-release.yml new file mode 100644 index 00000000..19127c36 --- /dev/null +++ b/.github/workflows/_template-emit-release.yml @@ -0,0 +1,85 @@ +name: notify-orchestrator + +# TEMPLATE — copy this file into any sibling repo's `.github/workflows/` and +# rename to `notify-orchestrator.yml`. The repo's `release.yml` (or whatever +# workflow tags releases) gains this step at its end. Whenever the repo tags +# a release, this workflow notifies the central orchestrator at +# `web4/.github/workflows/orchestrator.yml`, which then computes the set of +# downstream nodes that need a bump and dispatches to each. +# +# Required secret on this repo: +# ORCHESTRATOR_DISPATCH_TOKEN — repository_dispatch scope on web4. Prefer a +# GitHub App token; one App installation can cover every sibling so the +# secret is the same value everywhere. +# +# Placeholders to fill before adopting: +# — the package name as it appears in deps.json (e.g. "policy", +# "handshake", "libpilot"). Must match exactly. +# +# Example wiring at the end of a sibling's release.yml: +# +# notify: +# needs: release +# uses: ./.github/workflows/notify-orchestrator.yml +# with: +# node: policy +# secrets: +# ORCHESTRATOR_DISPATCH_TOKEN: ${{ secrets.ORCHESTRATOR_DISPATCH_TOKEN }} + +on: + workflow_call: + inputs: + node: + description: 'Node name in web4/.github/deps.json' + required: true + type: string + secrets: + ORCHESTRATOR_DISPATCH_TOKEN: + required: true + workflow_dispatch: + inputs: + node: + description: 'Node name in web4/.github/deps.json' + required: true + version: + description: 'Tag to notify about (e.g. v1.0.5). Defaults to github.ref_name when called from a release workflow.' + required: false + +jobs: + notify: + runs-on: ubuntu-latest + steps: + - name: Emit package-released + env: + TOKEN: ${{ secrets.ORCHESTRATOR_DISPATCH_TOKEN }} + NODE: ${{ inputs.node }} + VERSION: ${{ inputs.version || github.ref_name }} + run: | + if [ -z "$TOKEN" ]; then + echo "::warning::ORCHESTRATOR_DISPATCH_TOKEN unset — skipping notification." + exit 0 + fi + IS_PRE=false + case "$VERSION" in + *-rc*|*-beta*|*-alpha*) IS_PRE=true ;; + esac + + payload=$(jq -nc \ + --arg pkg "$NODE" \ + --arg ver "$VERSION" \ + --argjson pre "$IS_PRE" \ + '{event_type:"package-released", client_payload:{package:$pkg, version:$ver, is_prerelease:$pre}}') + + HTTP_CODE=$(curl -sS -o /tmp/resp -w '%{http_code}' \ + -X POST \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${TOKEN}" \ + "https://api.github.com/repos/TeoSlayer/pilotprotocol/dispatches" \ + -d "$payload") + + if [ "$HTTP_CODE" != "204" ]; then + echo "orchestrator notify failed (HTTP $HTTP_CODE):" + cat /tmp/resp + exit 1 + fi + echo "orchestrator notified: ${NODE} ${VERSION} (is_prerelease=${IS_PRE})" diff --git a/.github/workflows/orchestrator.yml b/.github/workflows/orchestrator.yml new file mode 100644 index 00000000..35924c5f --- /dev/null +++ b/.github/workflows/orchestrator.yml @@ -0,0 +1,204 @@ +name: orchestrator + +# Central release orchestrator. Any repo in the constellation that ships a +# release `repository_dispatch`es to this workflow with: +# +# { event_type: "package-released", +# client_payload: { package: "", version: "v1.2.3", is_prerelease: false } } +# +# The orchestrator reads `.github/deps.json` (the canonical dependency graph), +# computes the reverse-transitive closure of `package` — i.e. every node that +# imports the released one, directly or through a chain — and emits a +# `bump-upstream` dispatch to each one. The receivers (one per repo) decide +# whether to commit-to-main, open a PR, or skip based on their own bump_policy. +# +# Why centralize: +# - The dependency graph lives in exactly one place; no repo needs to know +# who its dependents are. Add or remove an edge by editing deps.json here. +# - Topological order is computed once, not negotiated by receivers. +# - One token (SHOCKWAVE_DISPATCH_TOKEN) covers every fan-out hop. +# +# Why deps.json over a fancier graph store: +# - Reviewable: changes show up as PR diffs. +# - Auditable: `git log` tells you when an edge was added. +# - No runtime infra: this workflow is the whole system. + +on: + repository_dispatch: + types: [package-released] + workflow_dispatch: + inputs: + package: + description: 'Upstream package node name (must exist in deps.json)' + required: true + version: + description: 'Upstream version (e.g. v1.10.6)' + required: true + is_prerelease: + description: 'true if the upstream tag is a prerelease' + required: false + default: 'false' + dry_run: + description: 'When true, print the closure but do not dispatch' + required: false + default: 'false' + +permissions: + contents: read + +jobs: + fan-out: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Resolve inputs + id: input + env: + DISPATCH_PKG: ${{ github.event.client_payload.package }} + DISPATCH_VER: ${{ github.event.client_payload.version }} + DISPATCH_PRE: ${{ github.event.client_payload.is_prerelease }} + MANUAL_PKG: ${{ inputs.package }} + MANUAL_VER: ${{ inputs.version }} + MANUAL_PRE: ${{ inputs.is_prerelease }} + MANUAL_DRYRUN: ${{ inputs.dry_run }} + run: | + if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then + echo "package=$MANUAL_PKG" >> "$GITHUB_OUTPUT" + echo "version=$MANUAL_VER" >> "$GITHUB_OUTPUT" + echo "is_pre=$MANUAL_PRE" >> "$GITHUB_OUTPUT" + echo "dry_run=$MANUAL_DRYRUN" >> "$GITHUB_OUTPUT" + else + echo "package=$DISPATCH_PKG" >> "$GITHUB_OUTPUT" + echo "version=$DISPATCH_VER" >> "$GITHUB_OUTPUT" + echo "is_pre=$DISPATCH_PRE" >> "$GITHUB_OUTPUT" + echo "dry_run=false" >> "$GITHUB_OUTPUT" + fi + + - name: Compute reverse-transitive closure + id: closure + env: + PKG: ${{ steps.input.outputs.package }} + run: | + python3 - <<'PY' > /tmp/closure.json + import json, sys, os + from collections import defaultdict, deque + + deps = json.load(open(".github/deps.json")) + pkg = os.environ["PKG"] + if pkg not in deps["nodes"]: + print(f"::error::package '{pkg}' not in deps.json", file=sys.stderr) + sys.exit(1) + + # Build child map: children[upstream] = [downstream, ...] + children = defaultdict(list) + for node, info in deps["nodes"].items(): + for up in info.get("depends_on", []): + children[up].append(node) + + # BFS by depth so we can emit a topologically-ordered list. + depth = {pkg: 0} + order = [] + q = deque([pkg]) + while q: + n = q.popleft() + for c in children.get(n, []): + # Use max depth across all paths to root — guarantees a + # node's upstream peers in the closure are all visited + # before it is. + new_d = depth[n] + 1 + if c not in depth or new_d > depth[c]: + depth[c] = new_d + q.append(c) + + targets = sorted( + (c for c in depth if c != pkg), + key=lambda c: (depth[c], c), + ) + + out = [] + for t in targets: + info = deps["nodes"][t] + out.append({ + "node": t, + "repo": info["repo"], + "depth": depth[t], + "bump_policy": info.get("bump_policy", {}), + "type": info.get("type", "unknown"), + }) + + json.dump({"upstream": pkg, "targets": out}, sys.stdout, indent=2) + PY + + cat /tmp/closure.json + echo "payload=$(jq -c . /tmp/closure.json)" >> "$GITHUB_OUTPUT" + + - name: Fan-out dispatch (or dry-run report) + if: steps.input.outputs.dry_run != 'true' + env: + TOKEN: ${{ secrets.SHOCKWAVE_DISPATCH_TOKEN }} + UPSTREAM: ${{ steps.input.outputs.package }} + VERSION: ${{ steps.input.outputs.version }} + IS_PRE: ${{ steps.input.outputs.is_pre }} + CLOSURE: ${{ steps.closure.outputs.payload }} + run: | + if [ -z "$TOKEN" ]; then + echo "::warning::SHOCKWAVE_DISPATCH_TOKEN unset — skipping fan-out." + echo "Action required: set the secret and re-run with workflow_dispatch." + exit 0 + fi + + summary="| node | depth | type | result |\n|---|---|---|---|\n" + # Iterate the closure JSON's targets array. + echo "$CLOSURE" | jq -c '.targets[]' | while read -r t; do + REPO=$(echo "$t" | jq -r '.repo') + NODE=$(echo "$t" | jq -r '.node') + DEPTH=$(echo "$t" | jq -r '.depth') + TYPE=$(echo "$t" | jq -r '.type') + STABLE_ONLY=$(echo "$t" | jq -r '.bump_policy.stable_only // false') + + # Per-receiver stable-only gate: orchestrator could skip the + # dispatch entirely when the upstream is a prerelease AND the + # receiver says stable_only. Saves the receiver an API call. + if [ "$IS_PRE" = "true" ] && [ "$STABLE_ONLY" = "true" ]; then + printf " ~ skip %-22s depth=%s (stable_only and upstream is prerelease)\n" "$NODE" "$DEPTH" + summary="${summary}| ${NODE} | ${DEPTH} | ${TYPE} | skipped (prerelease) |\n" + continue + fi + + payload=$(jq -nc \ + --arg upstream "$UPSTREAM" \ + --arg version "$VERSION" \ + --argjson pre "$IS_PRE" \ + '{event_type:"bump-upstream", client_payload:{upstream:$upstream, version:$version, is_prerelease:$pre}}') + + HTTP_CODE=$(curl -sS -o /tmp/resp -w '%{http_code}' \ + -X POST \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${TOKEN}" \ + "https://api.github.com/repos/${REPO}/dispatches" \ + -d "$payload") + + if [ "$HTTP_CODE" = "204" ]; then + printf " ✓ dispatched to %-25s depth=%s\n" "$REPO" "$DEPTH" + summary="${summary}| ${NODE} | ${DEPTH} | ${TYPE} | ✓ dispatched |\n" + else + printf " ✗ %-25s HTTP=%s\n" "$REPO" "$HTTP_CODE" + cat /tmp/resp + summary="${summary}| ${NODE} | ${DEPTH} | ${TYPE} | ✗ HTTP ${HTTP_CODE} |\n" + fi + done + + printf "## Orchestrator fan-out for %s %s\n\nUpstream is_prerelease: %s\n\n%s" \ + "$UPSTREAM" "$VERSION" "$IS_PRE" "$summary" >> "$GITHUB_STEP_SUMMARY" + + - name: Dry-run summary only + if: steps.input.outputs.dry_run == 'true' + env: + CLOSURE: ${{ steps.closure.outputs.payload }} + run: | + echo "## Dry-run closure for ${{ steps.input.outputs.package }}" \ + >> "$GITHUB_STEP_SUMMARY" + echo '```json' >> "$GITHUB_STEP_SUMMARY" + echo "$CLOSURE" | jq . >> "$GITHUB_STEP_SUMMARY" + echo '```' >> "$GITHUB_STEP_SUMMARY" From 92007005d07c738b0748c010f388170e89a20dc8 Mon Sep 17 00:00:00 2001 From: Teodor Calin Date: Thu, 28 May 2026 16:22:26 -0700 Subject: [PATCH 4/4] ci(orchestrator): auto-generated deps.json + central orchestrator + receiver templates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the hardcoded shockwave fan-out in release.yml with a single notify-orchestrator step that emits package-released. The orchestrator workflow reads .github/deps.json (now auto-generated by scripts/build-deps.py from real go.mod / package.json / Package.swift) and computes the reverse-transitive closure of the released package, then dispatches per-target bump-upstream events in topological order. Why this is better than the hardcoded fan-out: - Adding/removing a sibling no longer touches release.yml; just create or delete the repo and the next deps refresh picks it up. - The graph is always in sync with reality. scripts/build-deps.py parses live go.mod files; refresh-deps-graph.yml runs hourly and on repository_dispatch deps-touched. When the regen produces a diff it opens (or updates) auto/deps-graph-refresh PR. - Sibling-to-sibling edges that the hardcoded shockwave missed (rendezvous → handshake → libpilot, eventstream → dataexchange, etc.) are now caught by including indirect requires. - Stable-only policy is honored per-receiver inside the orchestrator (the bump dispatch is skipped when upstream is a prerelease AND the receiver wants stable only). Hand-maintained input is now only: .github/deps.policy.json — bump policy per node + freeloader markings + manual_nodes for repos with no manifest (homebrew-pilot, website, freeloaders). Files in this commit: scripts/build-deps.py generator (gh + raw fetch) .github/deps.json auto-output, 27 nodes .github/deps.policy.json hand-maintained sidecar .github/workflows/orchestrator.yml unchanged from prior commit .github/workflows/refresh-deps-graph.yml NEW — scheduled regen + PR .github/workflows/release.yml drop shockwave; add notify-orchestrator step .github/workflows/_template-emit-release.yml unchanged .github/workflows/_template-bump-upstream.yml unchanged Required secret: SHOCKWAVE_DISPATCH_TOKEN (repository_dispatch scope on every sibling + the orchestrator's own repo). --- .github/deps.json | 371 ++++++++++++++++------- .github/deps.policy.json | 35 +++ .github/workflows/refresh-deps-graph.yml | 83 +++++ .github/workflows/release.yml | 66 ++-- scripts/build-deps.py | 308 +++++++++++++++++++ 5 files changed, 722 insertions(+), 141 deletions(-) create mode 100644 .github/deps.policy.json create mode 100644 .github/workflows/refresh-deps-graph.yml create mode 100644 scripts/build-deps.py diff --git a/.github/deps.json b/.github/deps.json index 12de2df9..8760f4e3 100644 --- a/.github/deps.json +++ b/.github/deps.json @@ -1,174 +1,339 @@ { "$schema": "https://pilotprotocol.network/.well-known/deps.schema.json", - "version": 1, - "description": "Canonical dependency graph of every Pilot Protocol repo. Read by .github/workflows/orchestrator.yml to compute which downstream nodes need a bump when any package ships a release. This is the SINGLE source of truth — keep it in sync whenever a repo gains or drops an upstream dependency.", + "version": 3, + "_generated_by": "scripts/build-deps.py", + "_do_not_edit": "Regenerated on every push by .github/workflows/refresh-deps-graph.yml. Edit deps.policy.json to override bump_policy or add manual_nodes.", "nodes": { - "web4": { - "repo": "TeoSlayer/pilotprotocol", - "type": "hub", - "go_module": "github.com/TeoSlayer/pilotprotocol", - "depends_on": [] + "app-store": { + "repo": "pilot-protocol/app-store", + "type": "go-app", + "go_module": "github.com/pilot-protocol/app-store", + "depends_on": [ + "common" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } }, - "beacon": { "repo": "pilot-protocol/beacon", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/beacon", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/beacon", + "depends_on": [ + "common" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } + }, + "check-layers": { + "repo": "pilot-protocol/check-layers", + "type": "go-sibling", + "go_module": "github.com/pilot-protocol/check-layers", + "depends_on": [] + }, + "common": { + "repo": "pilot-protocol/common", + "type": "shared-types", + "go_module": "github.com/pilot-protocol/common", + "depends_on": [], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } + }, + "cosift": { + "repo": "pilot-protocol/cosift", + "type": "go-sibling", + "go_module": "github.com/pilot-protocol/cosift", + "depends_on": [] }, "dataexchange": { "repo": "pilot-protocol/dataexchange", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/dataexchange", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/dataexchange", + "depends_on": [ + "common", + "eventstream", + "web4" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } }, "eventstream": { "repo": "pilot-protocol/eventstream", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/eventstream", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/eventstream", + "depends_on": [ + "common" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } }, "examples": { "repo": "pilot-protocol/examples", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/examples", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "depends_on": [], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } }, "gateway": { "repo": "pilot-protocol/gateway", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/gateway", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/gateway", + "depends_on": [ + "common" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } }, "handshake": { "repo": "pilot-protocol/handshake", - "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/handshake", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "type": "go-plugin", + "go_module": "github.com/pilot-protocol/handshake", + "depends_on": [ + "common", + "rendezvous", + "trustedagents", + "web4" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true, + "_note": "Imports web4/pkg/daemon \u2014 daemon-plugin coupling. Bumps coordinated with web4." + } + }, + "homebrew-pilot": { + "repo": "TeoSlayer/homebrew-pilot", + "type": "package-tap", + "depends_on": [ + "web4" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } + }, + "libpilot": { + "repo": "pilot-protocol/libpilot", + "type": "ffi-fan-in", + "go_module": "github.com/pilot-protocol/libpilot", + "depends_on": [ + "common", + "handshake", + "policy", + "runtime", + "skillinject", + "trustedagents", + "web4" + ], + "bump_policy": { + "stable_only": true, + "open_pr_instead_of_main": true, + "_note": "FFI fan-in; bad bump cascades to all SDKs." + } }, "nameserver": { "repo": "pilot-protocol/nameserver", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/nameserver", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/nameserver", + "depends_on": [ + "common" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } + }, + "pilot-ca": { + "repo": "pilot-protocol/pilot-ca", + "type": "go-sibling", + "go_module": "github.com/pilot-protocol/pilot-ca", + "depends_on": [] }, "policy": { "repo": "pilot-protocol/policy", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/policy", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/policy", + "depends_on": [ + "common" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } }, "rendezvous": { "repo": "pilot-protocol/rendezvous", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/rendezvous", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/rendezvous", + "depends_on": [ + "beacon", + "common", + "web4" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } }, "runtime": { "repo": "pilot-protocol/runtime", - "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/runtime", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "type": "go-plugin", + "go_module": "github.com/pilot-protocol/runtime", + "depends_on": [ + "common", + "handshake", + "policy", + "trustedagents", + "web4" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true, + "_note": "Imports web4/pkg/daemon \u2014 daemon-plugin coupling. Bumps coordinated with web4." + } + }, + "sdk-node": { + "repo": "pilot-protocol/sdk-node", + "type": "sdk", + "depends_on": [ + "libpilot" + ], + "bump_policy": { + "stable_only": true, + "open_pr_instead_of_main": true + } + }, + "sdk-python": { + "repo": "pilot-protocol/sdk-python", + "type": "sdk", + "depends_on": [ + "libpilot" + ], + "bump_policy": { + "stable_only": true, + "open_pr_instead_of_main": true + } + }, + "sdk-swift": { + "repo": "pilot-protocol/sdk-swift", + "type": "sdk", + "depends_on": [ + "libpilot" + ], + "bump_policy": { + "stable_only": true, + "open_pr_instead_of_main": true + } }, "skillinject": { "repo": "pilot-protocol/skillinject", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/skillinject", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/skillinject", + "depends_on": [ + "common" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } }, "trustedagents": { "repo": "pilot-protocol/trustedagents", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/trustedagents", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/trustedagents", + "depends_on": [ + "common" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } }, "updater": { "repo": "pilot-protocol/updater", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/updater", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/updater", + "depends_on": [ + "common" + ], + "bump_policy": { + "stable_only": true, + "auto_commit_main": true + } }, - "webhook": { - "repo": "pilot-protocol/webhook", + "wallet": { + "repo": "pilot-protocol/wallet", "type": "go-sibling", - "go_module": "github.com/TeoSlayer/pilotprotocol/webhook", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "go_module": "github.com/pilot-protocol/wallet", + "depends_on": [ + "app-store" + ] }, - "app-store": { - "repo": "pilot-protocol/app-store", - "type": "go-app", - "go_module": "github.com/TeoSlayer/pilotprotocol/app-store/integration", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } + "web4": { + "repo": "TeoSlayer/pilotprotocol", + "type": "go-sibling", + "go_module": "github.com/TeoSlayer/pilotprotocol", + "depends_on": [ + "app-store", + "beacon", + "common", + "dataexchange", + "eventstream", + "gateway", + "handshake", + "nameserver", + "policy", + "rendezvous", + "runtime", + "skillinject", + "trustedagents", + "webhook" + ], + "bump_policy": { + "stable_only": true, + "open_pr_instead_of_main": true + }, + "_imports_pkg_daemon_via": [ + "handshake", + "runtime", + "libpilot" + ] }, - - "libpilot": { - "repo": "pilot-protocol/libpilot", - "type": "ffi-fan-in", - "go_module": "github.com/TeoSlayer/pilotprotocol/libpilot", - "depends_on": ["web4", "handshake", "policy", "runtime", "skillinject"], + "webhook": { + "repo": "pilot-protocol/webhook", + "type": "go-sibling", + "go_module": "github.com/pilot-protocol/webhook", + "depends_on": [ + "common", + "handshake", + "web4" + ], "bump_policy": { "stable_only": true, - "open_pr_instead_of_main": true, - "_note": "FFI fan-in feeds all 3 SDKs at build time — a bad bump cascades to every SDK. Use PR + CI gate." + "auto_commit_main": true } }, - - "sdk-node": { - "repo": "pilot-protocol/sdk-node", - "type": "sdk", - "package_manager": "npm", - "depends_on": ["libpilot"], - "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true } - }, - "sdk-python": { - "repo": "pilot-protocol/sdk-python", - "type": "sdk", - "package_manager": "pypi", - "depends_on": ["libpilot"], - "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true } - }, - "sdk-swift": { - "repo": "pilot-protocol/sdk-swift", - "type": "sdk", - "package_manager": "swiftpm", - "depends_on": ["libpilot"], - "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true } - }, - - "homebrew-pilot": { - "repo": "TeoSlayer/homebrew-pilot", - "type": "package-tap", - "depends_on": ["web4"], - "bump_policy": { "stable_only": true, "auto_commit_main": true } - }, "website": { "repo": "pilot-protocol/website", "type": "surface", - "depends_on": ["web4"], + "depends_on": [], "bump_policy": { "stable_only": false, "auto_commit_main": true, "_note": "Edge channel also publishes; manifest preserves latest_stable when upstream is prerelease." } - }, - - "cosift": { "repo": "pilot-protocol/cosift", "type": "freeloader", "depends_on": [] }, - "pilot-ca": { "repo": "pilot-protocol/pilot-ca", "type": "freeloader", "depends_on": [] }, - "wallet": { "repo": "pilot-protocol/wallet", "type": "indirect", "depends_on": ["app-store"] } + } } } diff --git a/.github/deps.policy.json b/.github/deps.policy.json new file mode 100644 index 00000000..1b8835a9 --- /dev/null +++ b/.github/deps.policy.json @@ -0,0 +1,35 @@ +{ + "_purpose": "Small hand-maintained sidecar that the auto-generator (scripts/build-deps.py) merges into the auto-derived deps.json. Holds the things that CANNOT be derived from manifests: bump policy, freeloader markings, friendly notes. Keep this file as small as possible.", + "nodes": { + "common": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "beacon": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "dataexchange": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "eventstream": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "gateway": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "handshake": { "bump_policy": { "stable_only": true, "auto_commit_main": true, "_note": "Imports web4/pkg/daemon — daemon-plugin coupling. Bumps coordinated with web4." } }, + "nameserver": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "policy": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "rendezvous": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "runtime": { "bump_policy": { "stable_only": true, "auto_commit_main": true, "_note": "Imports web4/pkg/daemon — daemon-plugin coupling. Bumps coordinated with web4." } }, + "skillinject": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "trustedagents":{ "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "updater": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "webhook": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "app-store": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "libpilot": { "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true, "_note": "FFI fan-in; bad bump cascades to all SDKs." } }, + "examples": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "web4": { "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true } }, + "sdk-node": { "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true } }, + "sdk-python": { "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true } }, + "sdk-swift": { "bump_policy": { "stable_only": true, "open_pr_instead_of_main": true } }, + "homebrew-pilot": { "bump_policy": { "stable_only": true, "auto_commit_main": true } }, + "website": { "bump_policy": { "stable_only": false, "auto_commit_main": true, "_note": "Edge channel also publishes; manifest preserves latest_stable when upstream is prerelease." } } + }, + "manual_nodes": { + "homebrew-pilot": { "repo": "TeoSlayer/homebrew-pilot", "depends_on": ["web4"] }, + "website": { "repo": "pilot-protocol/website", "depends_on": ["web4"] }, + "cosift": { "repo": "pilot-protocol/cosift", "depends_on": [] }, + "pilot-ca": { "repo": "pilot-protocol/pilot-ca", "depends_on": [] }, + "wallet": { "repo": "pilot-protocol/wallet", "depends_on": ["app-store"] } + } +} diff --git a/.github/workflows/refresh-deps-graph.yml b/.github/workflows/refresh-deps-graph.yml new file mode 100644 index 00000000..bbe16624 --- /dev/null +++ b/.github/workflows/refresh-deps-graph.yml @@ -0,0 +1,83 @@ +name: refresh-deps-graph + +# Auto-regenerates .github/deps.json from the live state of every Pilot +# Protocol repo's go.mod / package.json / Package.swift. The orchestrator +# (orchestrator.yml) reads deps.json to compute cascade closures, so +# keeping it in sync with reality is what makes the cascade reliable. +# +# Triggers: +# - schedule: hourly. Catches edges added to siblings between explicit pushes. +# - workflow_dispatch: manual rerun (e.g. just after merging a structural PR). +# - repository_dispatch deps-touched: every sibling's CI fires this when its +# go.mod changes. Faster than waiting for the hourly cron. +# +# When the regen produces a diff against the committed file, this workflow +# opens (or updates) a PR titled "chore(deps): refresh dependency graph" so +# the change is reviewable. Auto-commit-to-main is intentionally NOT enabled — +# changes to deps.json affect orchestrator behavior org-wide. +# +# Required token: +# GITHUB_TOKEN is enough for the read paths (gh repo list, raw file fetch +# on public repos). The PR-open step also uses GITHUB_TOKEN. + +on: + schedule: + - cron: '13 * * * *' + workflow_dispatch: + repository_dispatch: + types: [deps-touched] + +permissions: + contents: write + pull-requests: write + +jobs: + refresh: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Regenerate deps.json + env: + GH_TOKEN: ${{ github.token }} + run: | + python3 scripts/build-deps.py + if git diff --quiet .github/deps.json; then + echo "deps.json unchanged — nothing to do." + echo "CHANGED=0" >> "$GITHUB_ENV" + else + echo "deps.json changed; will open or update PR." + echo "CHANGED=1" >> "$GITHUB_ENV" + fi + + - name: Open / update PR + if: env.CHANGED == '1' + env: + GH_TOKEN: ${{ github.token }} + run: | + BRANCH="auto/deps-graph-refresh" + git config user.name "pilot-deps-bot" + git config user.email "deps-bot@pilotprotocol.network" + git checkout -B "$BRANCH" + git add .github/deps.json + git commit -m "chore(deps): refresh dependency graph" \ + -m "Regenerated by scripts/build-deps.py from the current go.mod / package.json / Package.swift of every Pilot Protocol repo. Review before merging — these edges drive orchestrator cascade decisions." + git push --force-with-lease origin "$BRANCH" + + # If a PR is already open, leave it (force-push updated it). + existing=$(gh pr list --head "$BRANCH" --json number -q '.[0].number' || true) + if [ -z "$existing" ]; then + gh pr create \ + --title "chore(deps): refresh dependency graph" \ + --body "Automated regeneration of \`.github/deps.json\` from current sibling manifests. See diff for any added/removed edges. + +This PR is auto-rebased on every schedule tick — closing it will block all subsequent refreshes; instead, merge or request changes." \ + --head "$BRANCH" \ + --base main + else + echo "Existing PR #$existing updated." + fi diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d0c991d7..2b739529 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -390,53 +390,43 @@ jobs: echo "manifest dispatch accepted (HTTP 204)" # ---------------------------------------------------------------------------- - # shockwave + # notify-orchestrator # - # Notify every package that derives from web4 (Homebrew formula + SDKs) that - # a new release exists. Each downstream repo runs its own bump workflow on - # receiving `repository_dispatch` event_type=upstream-release. + # The hardcoded shockwave fan-out has been superseded by the central + # orchestrator at .github/workflows/orchestrator.yml. Instead of release.yml + # knowing about every downstream consumer, it just announces "web4 shipped + # vX.Y.Z" and the orchestrator (which reads .github/deps.json) computes the + # reverse-transitive closure and dispatches per-target bumps. # - # Receivers (each must have a workflow listening for `upstream-release`): - # - pilot-protocol/homebrew-pilot → bump Formula/pilot.rb - # - pilot-protocol/sdk-node → bump pkg version + npm publish - # - pilot-protocol/sdk-python → bump pyproject + PyPI publish - # - pilot-protocol/sdk-swift → bump Package.swift binaryTarget - # - # Soft-fail per receiver: a missing token or a 404 on one repo MUST NOT block - # the others. The job summary at the end lists which targets succeeded so a - # missed dispatch is visible without grepping logs. + # Soft-fail: missing token emits a warning and exits 0; release flow not + # blocked. # ---------------------------------------------------------------------------- - shockwave: - name: Shockwave fan-out + notify-orchestrator: + name: Notify orchestrator needs: release runs-on: ubuntu-latest steps: - - name: Dispatch to downstream consumers + - name: Emit package-released env: - SHOCKWAVE_TOKEN: ${{ secrets.SHOCKWAVE_DISPATCH_TOKEN }} + TOKEN: ${{ secrets.SHOCKWAVE_DISPATCH_TOKEN }} TAG: ${{ github.ref_name }} IS_PRE: ${{ contains(github.ref_name, '-rc') || contains(github.ref_name, '-beta') }} run: | - if [ -z "$SHOCKWAVE_TOKEN" ]; then - echo "::warning::SHOCKWAVE_DISPATCH_TOKEN secret unset — skipping fan-out." + if [ -z "$TOKEN" ]; then + echo "::warning::SHOCKWAVE_DISPATCH_TOKEN unset — orchestrator not notified." exit 0 fi - summary="" - for repo in homebrew-pilot sdk-node sdk-python sdk-swift; do - payload=$(jq -nc --arg tag "$TAG" --argjson pre "$IS_PRE" \ - '{event_type:"upstream-release", client_payload:{tag:$tag, is_prerelease:$pre}}') - HTTP_CODE=$(curl -sS -o /tmp/resp -w '%{http_code}' \ - -X POST \ - -H "Accept: application/vnd.github+json" \ - -H "Authorization: Bearer ${SHOCKWAVE_TOKEN}" \ - "https://api.github.com/repos/pilot-protocol/${repo}/dispatches" \ - -d "$payload") - if [ "$HTTP_CODE" = "204" ]; then - summary="${summary} ✓ ${repo}\n" - else - summary="${summary} ✗ ${repo} (HTTP ${HTTP_CODE})\n" - echo "::warning::shockwave dispatch failed for ${repo}: HTTP ${HTTP_CODE}" - cat /tmp/resp - fi - done - printf "Shockwave fan-out summary:\n${summary}" >> "$GITHUB_STEP_SUMMARY" + payload=$(jq -nc --arg tag "$TAG" --argjson pre "$IS_PRE" \ + '{event_type:"package-released", client_payload:{package:"web4", version:$tag, is_prerelease:$pre}}') + HTTP_CODE=$(curl -sS -o /tmp/resp -w '%{http_code}' \ + -X POST \ + -H "Accept: application/vnd.github+json" \ + -H "Authorization: Bearer ${TOKEN}" \ + "https://api.github.com/repos/${GITHUB_REPOSITORY}/dispatches" \ + -d "$payload") + if [ "$HTTP_CODE" != "204" ]; then + echo "orchestrator notify failed (HTTP $HTTP_CODE):" + cat /tmp/resp + exit 1 + fi + echo "orchestrator notified: web4 ${TAG} (is_prerelease=${IS_PRE})" diff --git a/scripts/build-deps.py b/scripts/build-deps.py new file mode 100644 index 00000000..02eb1d22 --- /dev/null +++ b/scripts/build-deps.py @@ -0,0 +1,308 @@ +#!/usr/bin/env python3 +""" +build-deps.py — generate .github/deps.json from the actual go.mod, package.json, +and Package.swift of every Pilot Protocol repo. + +Replaces the hand-maintained deps.json. Run on a schedule and on every relevant +push (see refresh-deps-graph.yml). If the regenerated file differs from what's +checked in, the workflow opens a PR (or commits to main) so the graph cannot +silently drift. + +What's derived (no human action required): + - depends_on edges → parsed from each repo's go.mod require + replace lines, + plus a tiny inference for SDK→libpilot build edges. + - go_module path → from `module ...` in go.mod. + - type → heuristic (presence of go.mod = go-sibling, package.json + with `pilotprotocol-*` deps = sdk-node, etc.) + +What's NOT derived (lives in deps.policy.json sidecar): + - bump_policy (stable_only, auto_commit_main, open_pr_instead_of_main) + - freeloader marking (no edges in either direction by intent, not by accident) + - friendly notes / _changelog text + - manual overrides for cases the parser misses + +Usage: + scripts/build-deps.py # write .github/deps.json + scripts/build-deps.py --check # exit 1 if regen disagrees with checked-in + scripts/build-deps.py --org pilot-protocol --hub TeoSlayer/pilotprotocol +""" +from __future__ import annotations +import argparse +import json +import re +import subprocess +import sys +import urllib.request +from pathlib import Path +from typing import Iterable + +SCRIPT_DIR = Path(__file__).resolve().parent +REPO_ROOT = SCRIPT_DIR.parent +DEPS_FILE = REPO_ROOT / ".github" / "deps.json" +POLICY_FILE = REPO_ROOT / ".github" / "deps.policy.json" + +# Patterns that mark a require line as pointing at a Pilot Protocol module. +PILOT_MODULE_RE = re.compile( + r"^github\.com/(pilot-protocol/[a-zA-Z0-9._-]+|TeoSlayer/pilotprotocol(?:/[a-zA-Z0-9./_-]+)?)(?:/[a-zA-Z0-9./_-]+)?$" +) + +def run(cmd: list[str]) -> str: + return subprocess.check_output(cmd, text=True).strip() + +def list_org_repos(org: str) -> list[dict]: + """Use gh CLI to enumerate every public repo in the org.""" + raw = run(["gh", "repo", "list", org, "--json", "name,defaultBranchRef", "--limit", "200"]) + return json.loads(raw) + +def fetch_raw(owner: str, repo: str, ref: str, path: str) -> str | None: + """Fetch a single file from GitHub raw. Return None if 404.""" + url = f"https://raw.githubusercontent.com/{owner}/{repo}/{ref}/{path}" + try: + with urllib.request.urlopen(url, timeout=10) as r: + return r.read().decode("utf-8", errors="replace") + except Exception: + return None + +def parse_go_mod(text: str) -> dict: + """ + Extract module path and require + replace edges from a go.mod file body. + Returns: + { "module": "", + "requires": [(path, version, indirect_bool)], + "replaces": [(from_path, to_path_or_module)] } + """ + out: dict = {"module": None, "requires": [], "replaces": []} + in_require_block = False + in_replace_block = False + for raw_line in text.splitlines(): + line = re.sub(r"//.*", "", raw_line).strip() + if not line: + continue + if line.startswith("module "): + out["module"] = line.split()[1].strip('"') + continue + # require block + if line.startswith("require (") : + in_require_block = True; continue + if line.startswith("replace ("): + in_replace_block = True; continue + if line == ")": + in_require_block = in_replace_block = False; continue + # single-line require + if line.startswith("require "): + tokens = line.split()[1:] + if len(tokens) >= 2: + indirect = "indirect" in raw_line + out["requires"].append((tokens[0], tokens[1], indirect)) + continue + if in_require_block: + tokens = line.split() + if len(tokens) >= 2: + indirect = "indirect" in raw_line + out["requires"].append((tokens[0], tokens[1], indirect)) + continue + # replace handling + if line.startswith("replace "): + body = line[len("replace "):] + parts = [p.strip() for p in body.split("=>", 1)] + if len(parts) == 2: + src = parts[0].split()[0] + dst = parts[1].split()[0] + out["replaces"].append((src, dst)) + continue + if in_replace_block: + parts = [p.strip() for p in line.split("=>", 1)] + if len(parts) == 2: + src = parts[0].split()[0] + dst = parts[1].split()[0] + out["replaces"].append((src, dst)) + continue + return out + +def module_to_node(mod_path: str, name_map: dict[str, str]) -> str | None: + """Translate a Go import path into a node name from name_map. Returns None if not a Pilot module.""" + # Trim any trailing subpath: pilot-protocol/handshake/x → pilot-protocol/handshake + for prefix in ("github.com/pilot-protocol/", "github.com/TeoSlayer/"): + if mod_path.startswith(prefix): + tail = mod_path[len(prefix):] + # tail may contain a sub-path + top = tail.split("/", 1)[0] + if prefix.endswith("TeoSlayer/") and top == "pilotprotocol": + return name_map.get("__hub__") # web4 + return name_map.get(top) + return None + +def classify_repo(name: str, has_go_mod: bool, has_pkg_json: str | None, has_package_swift: bool) -> str: + if name == "common": + return "shared-types" + if name == "libpilot": + return "ffi-fan-in" + if name == "homebrew-pilot": + return "package-tap" + if name == "website": + return "surface" + if name == "sdk-node": + return "sdk" + if name == "sdk-python": + return "sdk" + if name == "sdk-swift": + return "sdk" + if name == "app-store": + return "go-app" + if name == "examples": + return "go-sibling" + if name in ("handshake", "runtime"): + return "go-plugin" + if has_go_mod: + return "go-sibling" + return "unknown" + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--org", default="pilot-protocol") + ap.add_argument("--hub", default="TeoSlayer/pilotprotocol", + help="Special-case repo that owns the daemon engine") + ap.add_argument("--ref", default="main", help="Branch/tag/sha to read from each repo") + ap.add_argument("--output", default=str(DEPS_FILE)) + ap.add_argument("--check", action="store_true", + help="Exit 1 if the generated file differs from the file on disk") + ap.add_argument("--policy", default=str(POLICY_FILE), + help="Sidecar JSON with bump_policy and freeloader markings") + args = ap.parse_args() + + # Load policy sidecar (small hand-maintained file). + policy: dict = {} + if Path(args.policy).exists(): + policy = json.loads(Path(args.policy).read_text()) + + print(f"# Enumerating repos in {args.org} ...", file=sys.stderr) + org_repos = list_org_repos(args.org) + names = [r["name"] for r in org_repos] + + # Always include the hub explicitly. + hub_owner, hub_repo = args.hub.split("/", 1) + + # name_map: top-level path → node name. Hub is sentinel "__hub__". + name_map = {n: n for n in names} + name_map["__hub__"] = "web4" + # Add an explicit "web4" entry (the node label for the hub repo) so it + # appears in the output even though its owner is different. + names.append("web4") + seen_web4 = True + + nodes: dict[str, dict] = {} + + def add_node(name: str, repo: str, has_go: bool, has_node: bool, has_swift: bool, + go_module: str | None, edges: list[str], extra: dict | None = None): + nodes[name] = { + "repo": repo, + "type": classify_repo(name, has_go, "package.json" if has_node else None, has_swift), + } + if go_module: + nodes[name]["go_module"] = go_module + nodes[name]["depends_on"] = sorted(set(e for e in edges if e and e != name)) + # Merge policy sidecar + if name in policy.get("nodes", {}): + nodes[name].update(policy["nodes"][name]) + if extra: + nodes[name].update(extra) + + # Walk each org repo + for r in org_repos: + name = r["name"] + ref = r.get("defaultBranchRef", {}).get("name") or args.ref + gomod = fetch_raw(args.org, name, ref, "go.mod") + pkgjson = fetch_raw(args.org, name, ref, "package.json") + pkgswift = fetch_raw(args.org, name, ref, "Package.swift") + + # Skip noise repos with no manifest at all (docs, configs, runbooks, + # workflow repo, etc.). Manual_nodes in policy can still bring them + # in later if they need explicit representation. + if name not in policy.get("nodes", {}) and not gomod and not pkgjson and not pkgswift: + continue + + edges: list[str] = [] + go_module = None + if gomod: + parsed = parse_go_mod(gomod) + go_module = parsed["module"] + # Include BOTH direct and indirect requires — sibling-to-sibling + # edges (runtime → rendezvous, webhook → trustedagents, etc.) + # commonly land in go.mod as // indirect because the compiler + # discovers them transitively through the import graph; the + # cascade still needs to fire when an indirect upstream ships. + for path, _ver, _indirect in parsed["requires"]: + node = module_to_node(path, name_map) + if node and node != name: + edges.append(node) + # Nested submodule (app-store/integration, examples/go) edges + # accrue to the parent node so a sibling push triggers any + # downstream that depends on its umbrella name. + for sub in ("integration/go.mod", "go/go.mod"): + nested = fetch_raw(args.org, name, ref, sub) + if nested: + p2 = parse_go_mod(nested) + for path, _ver, _indirect in p2["requires"]: + node = module_to_node(path, name_map) + if node and node != name: + edges.append(node) + # SDKs have implicit build-time edges to libpilot that are not in any manifest. + if name in ("sdk-node", "sdk-python", "sdk-swift"): + edges.append("libpilot") + add_node(name, f"{args.org}/{name}", + has_go=bool(gomod), + has_node=bool(pkgjson), + has_swift=bool(pkgswift), + go_module=go_module, edges=edges) + + # Hub: walk web4 separately. + hub_gomod = fetch_raw(hub_owner, hub_repo, args.ref, "go.mod") + if hub_gomod: + parsed = parse_go_mod(hub_gomod) + hub_edges: list[str] = [] + for path, _ver, indirect in parsed["requires"]: + if indirect: + continue + node = module_to_node(path, name_map) + if node and node != "web4": + hub_edges.append(node) + add_node("web4", args.hub, + has_go=True, has_node=False, has_swift=False, + go_module=parsed["module"], edges=hub_edges, + extra={"_imports_pkg_daemon_via": ["handshake", "runtime", "libpilot"]}) + + # Constellation freeloaders + manual surfaces (homebrew lives outside + # the org, website may not have manifests we can parse, etc.). The + # policy sidecar carries { node: {repo, depends_on} } for these. + for extra_name, manual in policy.get("manual_nodes", {}).items(): + if extra_name not in nodes: + nodes[extra_name] = { + "repo": manual["repo"], + "type": classify_repo(extra_name, False, None, False), + "depends_on": sorted(manual.get("depends_on", [])), + } + if extra_name in policy.get("nodes", {}): + nodes[extra_name].update(policy["nodes"][extra_name]) + + out = { + "$schema": "https://pilotprotocol.network/.well-known/deps.schema.json", + "version": 3, + "_generated_by": "scripts/build-deps.py", + "_do_not_edit": "Regenerated on every push by .github/workflows/refresh-deps-graph.yml. Edit deps.policy.json to override bump_policy or add manual_nodes.", + "nodes": dict(sorted(nodes.items())), + } + + out_text = json.dumps(out, indent=2) + "\n" + if args.check: + existing = Path(args.output).read_text() if Path(args.output).exists() else "" + if existing != out_text: + print("deps.json is out of date — regenerate with scripts/build-deps.py", file=sys.stderr) + return 1 + print("deps.json is up to date.", file=sys.stderr) + return 0 + Path(args.output).write_text(out_text) + print(f"wrote {args.output} ({len(nodes)} nodes)", file=sys.stderr) + return 0 + +if __name__ == "__main__": + sys.exit(main())