publish-smoke.yml

# Publish Smoke — the dynamic half of the #3091 prevention.
#
# 15.1.0 shipped with every fresh project's auth endpoints returning 500:
# the workspace's pnpm overrides (better-auth pinned to 1.7.0-rc.1) made all
# in-repo CI green, but overrides do NOT ship with published packages, so
# downstream installs resolved a dependency mix that was never tested here.
# The static gate is scripts/check-override-consistency.mjs (#3085 — override
# targets must be reflected in published manifests). This workflow is the
# dynamic gate: install the exact bits a user would get and drive the
# first-run flow (auth sign-up/sign-in/get-session + REST CRUD) for real.
#
# Two jobs, one driver script (scripts/publish-smoke.sh):
#
#   pack-smoke      SMOKE_MODE=pack — `pnpm pack` every publishable package
#                   (pack applies the same manifest rewrites as publish),
#                   scaffold a fresh project OUTSIDE the workspace, pin every
#                   publishable package — scoped and unscoped alike — to the
#                   tarballs via the project's own pnpm
#                   overrides, and smoke it. This is "what 15.1.0 would have
#                   failed": the release-candidate combination, no workspace
#                   overrides in sight.
#
#   registry-canary SMOKE_MODE=registry — weekly `npx create-objectstack@latest`
#                   against the real npm registry. Catches ^-range drift in the
#                   ecosystem (a transitive release) breaking ALREADY-published
#                   versions after the fact, AND — since the driver builds the
#                   scaffolded project — a published scaffold that no longer
#                   completes its own documented first run (#4902/#7644/#8677,
#                   each of which exited 0 through scaffold and install and
#                   failed only at build). Opens/refreshes an issue on failure.
#
# Trigger notes: the changesets release PR (changeset-release/main) is pushed
# with GITHUB_TOKEN, and GITHUB_TOKEN events never trigger other workflows —
# a plain `on: pull_request` / `on: push` would silently never run (the
# release PR has NO Actions checks today). So pack-smoke runs on
# `workflow_run` after each Release run completes (that's the moment
# changesets creates/updates the release branch), checks out the release
# branch if an open release PR exists, and reports the verdict back as a
# commit status on the branch head so it IS visible on the release PR.
# Not wired into normal PR CI on purpose: full build + pack + clean install
# is far too slow for the inner loop.
#
# That reporting path is also why there is a third job, resolve-guard (#5343):
# every verdict this workflow can produce reaches a human through that commit
# status, so a `resolve` that FAILS writes nothing at all — the release PR then
# looks exactly like "there was nothing to smoke". The status is not a required
# check (branch protection on main requires TypeScript Type Check, Build Core,
# Test Core and Dogfood Regression Gate), so an absent one blocks nothing and is
# simply invisible. resolve-guard turns that silence into an explicit red.
#
# "How long was publish broken?" is answered by walking the `publish-smoke /
# packed-tarballs` commit-status history on the release-branch head, never by
# scanning this workflow's run list, because the list interleaves the weekly
# registry canary (schedule-triggered, unrelated to any release candidate)
# with release-candidate smokes, and an unlabeled canary green reads as "the
# RC smoke passed" (#14190 — a six-day outage misread from exactly that).

name: Publish Smoke

run-name: ${{ github.event_name == 'schedule' && 'Registry canary (published latest) — scheduled' || github.event_name == 'workflow_dispatch' && format('Publish smoke — manual dispatch on {0}', github.ref_name) || format('Packed-tarball smoke (release candidate) — after Release run {0}', github.event.workflow_run.head_sha) }}

on:
  workflow_run:
    workflows: [Release]
    types: [completed]
  schedule:
    - cron: '47 4 * * 1' # weekly registry canary (Mon 04:47 UTC)
  workflow_dispatch:

permissions:
  contents: read

concurrency:
  group: publish-smoke-${{ github.event_name }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  resolve:
    name: Resolve target
    if: github.event_name != 'schedule'
    runs-on: ubuntu-latest
    timeout-minutes: 5
    outputs:
      run: ${{ steps.target.outputs.run }}
      ref: ${{ steps.target.outputs.ref }}
      report-sha: ${{ steps.target.outputs.report-sha }}
    steps:
      - name: Pick the ref to smoke
        id: target
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
            # Manual run: smoke whatever ref the run was dispatched on.
            echo "run=true" >> "$GITHUB_OUTPUT"
            echo "ref=${{ github.ref }}" >> "$GITHUB_OUTPUT"
            echo "report-sha=" >> "$GITHUB_OUTPUT"
            exit 0
          fi
          # workflow_run (a Release run finished): smoke the release branch
          # iff an open changesets release PR exists; otherwise skip.
          pr=$(gh pr list --repo "$GITHUB_REPOSITORY" \
            --head changeset-release/main --state open \
            --json headRefOid -q '.[0].headRefOid // empty')
          if [ -n "$pr" ]; then
            echo "run=true" >> "$GITHUB_OUTPUT"
            # Checkout the exact SHA the status is reported against, so a
            # concurrent force-push of the release branch can't skew the two.
            echo "ref=$pr" >> "$GITHUB_OUTPUT"
            echo "report-sha=$pr" >> "$GITHUB_OUTPUT"
            echo "Release PR open at $pr — smoking changeset-release/main"
          else
            echo "run=false" >> "$GITHUB_OUTPUT"
            echo "No open release PR — nothing to smoke"
          fi

  # resolve failing is NOT "nothing to smoke" — it is "we do not know". Say so
  # out loud (#5343). Without this job the two outcomes are indistinguishable
  # everywhere it matters: pack-smoke is skipped in both, and a skipped job reads
  # as green; and since this workflow runs on `workflow_run`, its own red run is
  # filed against the default branch, nowhere near the release PR a human is
  # about to merge.
  #
  # The alternative shape — #4928's `!cancelled() && ... != 'false'`, i.e. "when
  # in doubt, run everything" — is wrong HERE, and that is why the decision was
  # taken per workflow rather than copied. ci.yml's filter job defaults its
  # outputs to "run it" (`|| 'true'`); resolve does not, and it also computes the
  # `ref` to check out. Running on doubt would check out an EMPTY ref (the
  # workflow_run default), spend 45 minutes smoking whatever that resolves to,
  # and report the verdict as if it were about the release candidate. A false
  # green on release integrity is worse than no answer.
  resolve-guard:
    name: Resolve target failed — the candidate was NOT smoked
    needs: resolve
    if: always() && needs.resolve.result == 'failure'
    runs-on: ubuntu-latest
    timeout-minutes: 5
    permissions:
      contents: read
      statuses: write
    steps:
      - name: Report the gap on the release PR, then fail
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          # Best-effort, and deliberately a SECOND lookup rather than a reuse of
          # resolve's outputs: resolve failed, so its outputs are empty — that is
          # the whole problem. Same query as resolve's (`--head
          # changeset-release/main`); keep the two in step if that convention
          # ever changes. Whatever this lookup does, the job still fails below,
          # so the fallback can never turn into a silent pass.
          release_head=$(gh pr list --repo "$GITHUB_REPOSITORY" \
            --head changeset-release/main --state open \
            --json headRefOid -q '.[0].headRefOid // empty') || release_head=''
          if [ -n "$release_head" ]; then
            gh api "repos/$GITHUB_REPOSITORY/statuses/$release_head" \
              -f state=failure -f context='publish-smoke / packed-tarballs' \
              -f description='Target resolution failed — the candidate was never smoked' \
              -f target_url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \
              || echo "::warning::could not write the commit status on $release_head"
          else
            echo "::warning::no open changeset-release/main PR found — the failure is reported here only"
          fi
          echo "::error title=Publish smoke did not run::resolve failed, so the release candidate was never smoked. This is NOT the same as 'nothing to smoke' — re-run the workflow, or check the release PR by hand before merging."
          exit 1

  pack-smoke:
    name: Packed-tarball smoke (release candidate)
    needs: resolve
    # success() is spelled out rather than left to GitHub's implicit wrapper
    # (#5343): an `if:` naming no status function is silently wrapped in one, so
    # "resolve failed" and "resolve said don't run" both arrive here as the same
    # skip. The behaviour is unchanged — smoke only a resolve that SUCCEEDED and
    # said run — but it is now written down, and resolve-guard above states the
    # failure case that this condition deliberately does not cover.
    if: success() && needs.resolve.outputs.run == 'true'
    runs-on: ubuntu-latest
    timeout-minutes: 45
    permissions:
      contents: read
      statuses: write
    steps:
      - name: Mark pending on the release PR head
        if: needs.resolve.outputs.report-sha != ''
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          gh api "repos/$GITHUB_REPOSITORY/statuses/${{ needs.resolve.outputs.report-sha }}" \
            -f state=pending -f context='publish-smoke / packed-tarballs' \
            -f description='Installing the release candidate into a fresh project…' \
            -f target_url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"

      - name: Checkout repository
        uses: actions/checkout@v7
        with:
          ref: ${{ needs.resolve.outputs.ref }}

      - name: Setup Node.js
        uses: actions/setup-node@v7
        with:
          node-version: '22'

      - name: Setup pnpm
        uses: ./.github/actions/setup-pnpm

      - name: Get pnpm store directory
        shell: bash
        run: echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV

      - name: Setup pnpm cache
        uses: actions/cache@v6
        with:
          path: ${{ env.STORE_PATH }}
          key: ${{ runner.os }}-pnpm-store-v3-${{ hashFiles('**/pnpm-lock.yaml') }}
          restore-keys: |
            ${{ runner.os }}-pnpm-store-v3-

      - name: Setup turbo cache
        uses: actions/cache@v6
        with:
          path: .turbo/cache
          key: ${{ runner.os }}-turbo-${{ github.job }}-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-turbo-${{ github.job }}-
            ${{ runner.os }}-turbo-

      - name: Install dependencies
        run: pnpm install --frozen-lockfile

      - name: Build
        run: pnpm run build

      - name: Publish smoke (packed tarballs)
        run: bash scripts/publish-smoke.sh

      - name: Report verdict to the release PR head
        if: always() && needs.resolve.outputs.report-sha != ''
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          if [ "${{ job.status }}" = "success" ]; then
            state=success; desc='Fresh install of the release candidate: auth + CRUD green'
          else
            state=failure; desc='Release candidate fails a fresh install — see the run log'
          fi
          gh api "repos/$GITHUB_REPOSITORY/statuses/${{ needs.resolve.outputs.report-sha }}" \
            -f state="$state" -f context='publish-smoke / packed-tarballs' \
            -f description="$desc" \
            -f target_url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"

  registry-canary:
    name: Registry canary (published latest)
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    timeout-minutes: 25
    permissions:
      contents: read
      issues: write
    steps:
      - name: Checkout repository
        uses: actions/checkout@v7

      - name: Setup Node.js
        uses: actions/setup-node@v7
        with:
          node-version: '22'

      - name: Publish smoke (npm registry)
        run: SMOKE_MODE=registry bash scripts/publish-smoke.sh

      - name: Open or refresh the canary issue
        # Scheduled runs have no human watching — surface the breakage as an
        # issue (deduped: one open issue, refreshed with a comment per failure).
        if: failure() && github.event_name == 'schedule'
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          run_url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
          title="Registry canary failed: fresh npx create-objectstack install is broken"
          existing=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open \
            --search "\"$title\" in:title" --json number -q '.[0].number // empty')
          body=$(printf 'The weekly publish-smoke registry canary failed: a fresh `npx create-objectstack@latest` project no longer completes the first-run path against the npm registry — scaffold, npm install, npm run build, then auth + REST CRUD.\n\nThis job installs PUBLISHED artifacts, so a fix already merged to `main` does NOT clear it — only a release does. Check whether the range or template at fault is already fixed on `main` before opening new work.\n\nRead the run log to see WHICH step failed, because these classes have different owners. **Read the boot log before the probes** — a failed probe is usually a consequence, not the defect:\n\n- **a `failed to load` WARN in the boot log** — a dependency-range problem, and the FIRST thing to check whenever an auth or CRUD probe fails. Specimen: `⚠ AuthPlugin failed to load: The requested module @better-auth/core/db does not provide an export named createLocalAccountIssuer`. A static ESM named import of a missing export is a link-time SyntaxError, so the plugin never loads AT ALL and every later symptom follows from that one failure — core service missing, `sys_*` tables never created, sharing rules never seeded, and the probe failure this job finally exits on. Go to the DECLARED dependency range, NOT to the probe, and establish whether the vendor REMOVED or RENAMED the symbol: the two have different fixes. Reproduce it without waiting for a release using the vendor export-contract check in its resolve mode, which the daily Validate Dependencies workflow already runs: it installs every version each declared range admits and checks that export surface against the symbols we import.\n- **npm run build failed** — the published scaffold is broken on its own second command. This is the #4902/#7644/#8677 class; the fix is in create-objectstack or the template it ships.\n- **an auth or CRUD probe failed over a CLEAN boot log** — this is the #3091 class hitting ALREADY-published versions (e.g. a transitive dependency released into a ^ range).\n\n⚠ `✓ Server is ready` and the plugin count print on a DEGRADED boot too (#16630), so neither is evidence that the boot was healthy and neither narrows the branches above.\n\nRun log: %s\n' "$run_url")
          if [ -n "$existing" ]; then
            gh issue comment "$existing" --repo "$GITHUB_REPOSITORY" \
              --body "Still failing as of $run_url"
          else
            gh issue create --repo "$GITHUB_REPOSITORY" --title "$title" --body "$body"
          fi