live-e2e.yml

# Live E2E — the console driven against a REAL ObjectStack backend (#2835).
#
# What runs: an allowlist of e2e/live specs (the list is whatever the
# `test:e2e:live:ci` script in package.json names — that script is the single
# source of truth; do not re-enumerate the specs here, a hand-copied list
# drifts on every promotion) against `objectstack dev` booted from PUBLISHED
# @objectstack/* packages serving the showcase app. Every PR therefore
# smoke-tests "this console x the released backend" — the class of bug only
# a real browser against a real backend can catch (framework#3528: a
# lazily-loaded widget's suspension unwound to the host's route boundary and
# tore down the flow dialog; zero unit test could see it).
#
# ⚠️ INFORMATIONAL, NON-REQUIRED lane. What keeps it non-blocking is what it
# is NOT wired into, and ⛔ never a `continue-on-error:` on the job:
#   - it is not in the required-check set — ⛔ do not add this job to required
#     checks until the nightly record proves the lane stable; and
#   - it declares no `merge_group` trigger, so a queue build never waits on it
#     and it cannot eject an unrelated pull request from the queue.
# (objectstack#4850 is the prior art for why a new lane must prove itself
# outside the merge gate first.)
#
# ⛔ Do NOT re-add `continue-on-error: true` to this job (objectui#8084). It
# bought none of the non-blocking above, and it cost this lane the one reading
# a monitor, a digest or an agent takes cheaply. Measured on the 2026-09-06
# nightly, while the job still carried the flag:
#
#     step 7 `Start ObjectStack backend`    failure
#     job 101442890465                      failure
#     check run `Live E2E (informational)`  failure   <- the flag did not move it
#     check suite 92170955546               success   <- asserts the negation
#     workflow run 34017174769              success   <- asserts the negation
#
# The flag moved neither the job nor its check run — the two readings a merge
# gate and a reviewer actually consult — and inverted only the aggregate. So an
# outage detector reported `success` for a day while the E2E had not executed
# at all, which is how the backend outage carded as objectui#8084's defect ①
# ran unseen. Without the flag the run conclusion simply agrees with the job it
# contains, and the lane stays exactly as advisory as it was.
#
# Failures surface as a red step, a red job, a red run, a job summary and the
# `live-e2e-artifacts` upload. That upload carries `test-results/` (the
# failure screenshots and videos `use.screenshot` / `use.video` write) and the
# two server logs — it does NOT carry a Playwright HTML report, because
# `playwright.live.config.ts` declares `reporter: [['list']]` and the `list`
# reporter writes to stdout only (objectui#8238). The durable record of what
# the specs actually did is the `Run live E2E allowlist` step's own log.
#
# Growing the allowlist: add specs to `test:e2e:live:ci` in package.json a
# few at a time, only after they prove flake-free here — do not switch all
# 20+ live specs on at once and inherit whatever flake exists (#2835's
# sequencing note).
#
# Backend pins live in e2e/live/ci/backend.env — the published package
# version MUST match the @objectstack/spec version in pnpm-lock.yaml (a
# mismatched pair proves nothing; bump both in the same PR).
name: Live E2E

on:
  pull_request:
    branches: [main, develop]
    paths-ignore:
      - '**/*.md'
      - 'content/**'
      - 'docs/**'
      - 'apps/site/**'
      - '.changeset/**'
  # Nightly on main: accumulates the stability record that decides when the
  # lane may be promoted to a required check.
  schedule:
    - cron: '30 6 * * *'
  workflow_dispatch:

concurrency:
  group: live-e2e-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

permissions:
  contents: read

jobs:
  live-e2e:
    name: Live E2E (informational)
    runs-on: ubuntu-latest
    timeout-minutes: 40
    # ⛔ There is deliberately no `continue-on-error:` here, and ⛔ nothing may
    # add one back: it never made this job or its check run any less red, it
    # only made the workflow-run conclusion contradict them (objectui#8084).
    # The header carries the measurement.

    # ⚠️ Do NOT add a `runner.*` expression to this job-level `env:` block.
    # The `runner` context does not exist yet when job-level env is evaluated
    # (GitHub's "contexts availability" table allows only github / needs /
    # strategy / matrix / vars / secrets / inputs here). One `${{ runner.temp }}`
    # in this block fails the whole FILE at validation time — 0 jobs, 0s,
    # "startup failure" — and `continue-on-error` cannot soften it because no
    # job is ever created. Need a runner path? Use the `$RUNNER_TEMP`
    # environment variable inside a `run:` step, or a step-level `env:` /
    # `with:` (both may read `runner`). See the LIVE_BACKEND_DIR step below.
    env:
      LIVE_BACKEND_PORT: '4010'
      LIVE_API_URL: http://localhost:4010
      LIVE_APP_URL: http://localhost:5190

    steps:
      - name: Checkout code
        uses: actions/checkout@v7
        with:
          submodules: true

      - name: Enable Corepack and download the pinned pnpm
        run: bash scripts/ci-setup-pnpm.sh

      - name: Setup Node.js
        uses: actions/setup-node@v7
        with:
          node-version: '22.x'
          cache: 'pnpm'

      - name: Install dependencies
        run: pnpm install --frozen-lockfile

      # The backend fixture (showcase metadata + published @objectstack/*
      # node_modules) only changes when the pins do — cache it on the pin file.
      # start-backend.sh's stamp check makes a cache hit skip clone + install.
      #
      # ── Cache bookkeeping cannot void a recorded verdict (objectui#7048) ──
      # THE ORDERING, carried here from the type-check split in `ci.yml`
      # (objectui#6577, PR #7047) because it is what a future reader needs in
      # order to judge whether these steps may be touched: the verdict is
      # recorded by the checking steps; everything after them is bookkeeping,
      # and bookkeeping must never discard an answer the gate already produced.
      #
      # Why the SPLIT rather than a timeout on one step: combined
      # `actions/cache` declares `main: dist/restore/index.js` plus
      # `post: dist/save/index.js`, so its save is a step the RUNNER generates
      # at job end (`Post Cache live backend fixture`). No workflow syntax
      # attaches `timeout-minutes` or `continue-on-error` to a generated post
      # step, so an upload stall runs this job into `timeout-minutes: 40` and
      # it reports `cancelled` — a lane that says nothing at all, which for an
      # informational lane whose only product is a stability record is the
      # worst outcome available. Measured once, on `ci.yml`'s type-check cache:
      # a 1-second save took 789s (objectui#6577). `actions/cache`'s own
      # `save-always` deprecation text points at this same split.
      #
      # ⛔ Raising `timeout-minutes: 40` is the ruled-out non-fix: a larger
      # ceiling only buys a longer hang and still ends in `cancelled`.
      #
      # This job has TWO caches, so it gets two bounds. Together they can
      # consume at most 5 + 8 = 13 minutes of the 40-minute ceiling, against a
      # verdict path that measured 2m25s end to end (job 100097466897) — so
      # even both bookkeeping halves stalling at once cannot reach the ceiling.
      #
      # The restore halves are deliberately left UNBOUNDED: a restore stall
      # fails BEFORE any verdict exists — a lane that did not run, which is
      # honest — rather than a recorded verdict discarded.
      - name: Restore live backend fixture
        id: live-backend-cache
        uses: actions/cache/restore@v6
        with:
          path: ${{ runner.temp }}/live-backend
          key: live-backend-${{ runner.os }}-${{ hashFiles('e2e/live/ci/backend.env') }}

      - name: Start ObjectStack backend (published packages)
        # Step-level env — `runner` IS available here (unlike job-level env).
        # Must resolve to the same path as the cache step's `path:` above and
        # the artifact upload's `path:` below, or the cache silently stops
        # hitting; keep the three in sync.
        env:
          LIVE_BACKEND_DIR: ${{ runner.temp }}/live-backend
        run: bash e2e/live/ci/start-backend.sh

      - name: Build console
        # Absolute base so `vite preview`'s SPA fallback serves deep links
        # (/apps/showcase_app/…) with resolvable asset URLs — the default
        # relative base ('./') 404s them (same reason ci.yml's e2e job pins
        # VITE_BASE_PATH). Vite, not the package build: the SPA bundle is the
        # only artifact the tests consume.
        env:
          VITE_BASE_PATH: /
        run: pnpm --filter @object-ui/console exec vite build

      - name: Serve console (vite preview)
        # `preview.proxy` defaults to `server.proxy`, so /api is proxied to
        # DEV_PROXY_TARGET just like the dev server would.
        working-directory: apps/console
        run: |
          DEV_PROXY_TARGET="$LIVE_API_URL" nohup pnpm exec vite preview --port 5190 --strictPort \
            > "$RUNNER_TEMP/console-preview.log" 2>&1 &
          echo $! > "$RUNNER_TEMP/console-preview.pid"
          for i in $(seq 1 60); do
            curl -sf -o /dev/null "$LIVE_APP_URL" && exit 0
            sleep 1
          done
          echo "console preview never became ready" >&2
          tail -50 "$RUNNER_TEMP/console-preview.log" >&2
          exit 1

      - name: Get Playwright version
        id: playwright-version
        run: |
          # This pipeline's exit status is load-bearing: the value it produces
          # becomes the Playwright browser cache key in the next step. Without
          # `pipefail` a failing `pnpm list` is masked by `jq`, and `jq -r`
          # prints the string `null` and exits 0 when the field is missing — so
          # either failure used to yield a *successful* step and a key that had
          # silently degraded to `playwright-Linux-` / `playwright-Linux-null`.
          # That wrong bucket is stable, so two Playwright versions can share
          # one cache entry and restore a stale browser (objectui#6231).
          # The same block is duplicated verbatim in `ci.yml` and `live-e2e.yml`
          # — keep them byte-identical so they stay greppable as a pair.
          set -eo pipefail
          if ! version=$(pnpm list @playwright/test --depth=0 --json | jq -r '.[0].devDependencies["@playwright/test"].version'); then
            echo "::error::Reading the @playwright/test version failed (pnpm list --json | jq). Refusing to write a Playwright browser cache key from it."
            exit 1
          fi
          if [ -z "$version" ] || [ "$version" = "null" ]; then
            echo "::error::Could not resolve the @playwright/test version (got: '${version}'). Refusing to write an empty or null version into the Playwright browser cache key."
            exit 1
          fi
          echo "version=$version" >> "$GITHUB_OUTPUT"

      # The second of this job's two caches; the split, the ordering sentence
      # and the ruled-out non-fix are all documented at `Restore live backend
      # fixture` above (objectui#7048). Same shape here: restore in place and
      # unbounded, save last and bounded.
      - name: Restore Playwright browsers
        uses: actions/cache/restore@v6
        id: playwright-cache
        with:
          path: ~/.cache/ms-playwright
          key: playwright-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}

      # Same provisioning shape as `ci.yml`'s `Build & E2E`, and it carries the
      # same exposure: `--with-deps` / `install-deps` run `apt-get update` with
      # nothing bounding it. That is what hung `Build & E2E` three times on
      # 2026-08-19 for the full job ceiling (objectui#5304); this job's ceiling
      # is 40 minutes, so the same mirror stall would cost even more here. The
      # browsers come from the cache step above, and the launch probe in the
      # script is what proves a browser exists. See the script header for the
      # measured incident log.
      - name: Install Playwright browsers
        if: steps.playwright-cache.outputs.cache-hit != 'true'
        run: pnpm exec playwright install chromium

      - name: Ensure Chromium can launch
        run: bash scripts/ensure-chromium-ready.sh

      - name: Run live E2E allowlist
        run: pnpm test:e2e:live:ci

      - name: Report failure in job summary
        if: failure()
        run: |
          {
            echo "## Live E2E failed (informational lane — does not block merge)"
            echo ""
            echo "Backend: published \`@objectstack/*\` per \`e2e/live/ci/backend.env\`."
            echo "What the specs did is in this job's \`Run live E2E allowlist\` step log (the \`list\` reporter writes to stdout; there is no HTML report to open)."
            echo "See the \`live-e2e-artifacts\` upload for the failure screenshots and videos (\`test-results/\`) and the two server logs."
            echo "If this failure reproduces on re-run it is a real console x backend integration bug — treat it as such even though the lane cannot block your merge."
          } >> "$GITHUB_STEP_SUMMARY"

      # ⛔ Do not add `playwright-report/` back to this glob without changing
      # `playwright.live.config.ts` in the same PR. That config declares
      # `reporter: [['list']]`; the `list` reporter writes to stdout and no
      # reporter in it writes `playwright-report/`, so the path matched nothing
      # in EVERY outcome — measured on a passing and on a failing run, absent
      # both times — while this workflow's header and
      # `content/docs/guide/ci-cd-pipeline.md` promised readers a report they
      # could open (objectui#8238). The reporter list is the one line that
      # decides it, and the claim is pinned in
      # `scripts/__tests__/ci-cd-pipeline-doc.test.ts`.
      #
      # ⚠️ This step is gated on `failure()`, so on a GREEN lane it does not run
      # and there is NO artifact at all. Any acceptance criterion phrased over
      # this artifact's contents is therefore readable only on a run that
      # failed; a green run leaves nothing to count (objectui#8238).
      - name: Upload failure artefacts and server logs
        uses: actions/upload-artifact@v7
        if: ${{ !cancelled() && failure() }}
        with:
          name: live-e2e-artifacts
          path: |
            test-results/
            ${{ runner.temp }}/live-backend/backend.log
            ${{ runner.temp }}/console-preview.log
          retention-days: 14

      # The bookkeeping halves of the two splits documented at the restore
      # steps above, placed HERE — last in the job — because that is exactly
      # where the post phases they replace already ran. They are ordered
      # Playwright-then-fixture to preserve the observed post-phase order
      # (post steps run in reverse registration order, so `Post Cache
      # Playwright browsers` ran before `Post Cache live backend fixture` —
      # job 100097466897). Nothing about what this lane reports moves with them.
      #
      # `timeout-minutes: 8` on the browser cache is DERIVED FOR THIS CACHE,
      # not inherited from the type-check site's 5 (objectui#7048 fences that
      # explicitly): it is Playwright's browser binaries at ~269 MB. Restores
      # of that exact archive measured 5s here (job 100097466897) and 3s in
      # `ci.yml`'s `Build & E2E` (jobs 100096569775, 100094597323) ⇒ ~54-90
      # MB/s in the download direction; a save is only observable after a
      # version bump (an exact key hit skips it, and `Post Cache Playwright
      # browsers` measured 0s on every sampled run), so the honest save is
      # bounded from the restore rather than read directly — said plainly
      # rather than dressed up as a direct measurement. A deliberately
      # pessimistic 20x penalty on the slowest observed restore gives ~1m40s;
      # 8 minutes is ~4.8x that and demands only 0.56 MB/s sustained.
      #
      # `timeout-minutes: 5` on the fixture cache is derived from THIS site's
      # own measurement: its save measured 8s on 2026-09-02 (`Post Cache live
      # backend fixture`, job 100097466897 — a real save, not a skip: the
      # restore missed, which is why `Start ObjectStack backend` did the full
      # clone + install). 5 minutes is ~37x that.
      #
      # `continue-on-error: true` on both is the other half, and without it the
      # bounds would only trade a `cancelled` lane for a red one. A cache that
      # failed to upload costs the next run some time; it says nothing about
      # the console-x-backend integration this lane exists to observe.
      #
      # Behaviour preserved, spelled out so the equivalence is checkable:
      #   - `cache-hit != 'true'` reproduces the combined action's own "exact
      #     hit on the primary key ⇒ do not save" skip.
      #   - neither condition names a status function, so the implicit
      #     `success()` still applies — matching `post-if: success()`. In
      #     particular a failed `Run live E2E allowlist` skips both saves, just
      #     as the post phases skipped.
      #   - same `path` and same `key` as the matching restore step.
      - name: Save Playwright browsers
        if: steps.playwright-cache.outputs.cache-hit != 'true'
        uses: actions/cache/save@v6
        timeout-minutes: 8
        continue-on-error: true
        with:
          path: ~/.cache/ms-playwright
          key: playwright-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}

      - name: Save live backend fixture
        if: steps.live-backend-cache.outputs.cache-hit != 'true'
        uses: actions/cache/save@v6
        timeout-minutes: 5
        continue-on-error: true
        with:
          path: ${{ runner.temp }}/live-backend
          key: live-backend-${{ runner.os }}-${{ hashFiles('e2e/live/ci/backend.env') }}

      # Ephemeral runner — no teardown needed; stop-backend.sh exists for
      # local runs of the same scripts.