Skip to content

test(ci): prove the cooldown rejects a freshly published release - #598

Draft
guyofeck wants to merge 2 commits into
mainfrom
ci/cooldown-proof
Draft

test(ci): prove the cooldown rejects a freshly published release#598
guyofeck wants to merge 2 commits into
mainfrom
ci/cooldown-proof

Conversation

@guyofeck

Copy link
Copy Markdown
Contributor

Stacked on #597base that PR, not main. This is the test @dimar asked for in place of a code review:

i think the best "review" would be a test. Can we stack another PR on top of that and check if installing a fresh-package is rejected? I used npmjs.com/package/electron-nightly for always "fresh" versions

Adds cooldown-check.yml + check_cooldown.sh, which probes the real bunfig.toml setting in a temp project. Three cases, because an exit code alone can't tell a working guardrail from a broken probe:

Case Must hold
control — bun add lodash succeeds, so a red result means the cooldown fired, not that the probe broke
floating — bun add electron-nightly refused, or resolves to a version ≥ 7 days old — asserted against the registry's publish time, so silently installing a 1-day-old version can't pass
exact pin — bun add electron-nightly@<newest> refused. This is the bypass that matters: a PR pinning an exact fresh version

The cooldown is read out of bunfig.toml rather than hardcoded, so the test can't drift from the setting it verifies.

No gateway in this workflow, deliberately. Gatewayed, embargo would reject the fresh release itself and the run would prove nothing about Bun — which is the guardrail #597 relies on. check_wix_proxy_steps.py gains a third exemption and now records why each exists: the frozenset becomes a path -> reason dict, printed per exemption on success, so no exemption can be added without stating its justification.

No third-party actions, per the policy that landed with Actions being re-enabled: github_owned_allowed with empty patterns_allowed, and sha_pinning_required: true. So oven-sh/setup-bun isn't usable — Bun installs from a run step, and actions/checkout is pinned to 3d3c42e5 (v7.0.1).

Status

  • bash -n clean; bunfig parsing smoke-tested (604800 → 7 days)
  • check_wix_proxy_steps.py: 12/12 gatewayed jobs, all 3 exempt jobs verified to abstain; 13/13 unit tests
  • Not yet run in CI — pushing this PR should trigger it via the pull_request path filter

Heads-up for @dimar

The rest of the repo can't run under the new policy yet: lint, test, typecheck, knip and both publish workflows all depend on oven-sh/setup-bun, which the empty patterns_allowed blocks, and 15 of 16 uses: refs are still tag-pinned. This workflow was written to need neither, so it can run today — but the others need either allowlist entries or the same curl-install treatment. Which do you prefer?


🤖 Generated with Claude Code

guyofeck and others added 2 commits August 11, 2026 15:00
Implements the interim policy Dima Ryskin set out for OSS repos, so GitHub
Actions can be re-enabled for base44/cli at the org level:

  1. enforce lock-files on all node builds, so the build never overrides one
  2. enforce a package manager that supports a minimal-age directive
  3. use embargo in non-publish workflow tasks

Bun satisfies point 2 natively, so no package manager migration is needed:

    bun install --minimum-release-age=<val>
        Only install packages published at least N seconds ago (security feature)

bunfig.toml sets `minimumReleaseAge = 604800` (7 days; Bun takes SECONDS, unlike
npm's min-release-age, which takes days). Point 1 was already satisfied by
`bun install --frozen-lockfile`, which is what every workflow runs.

That choice matters operationally, not just aesthetically: Wix-managed machines
block registry.npmjs.org at the network-extension layer, so no developer here can
regenerate an npm lockfile without routing through the embargo gateway or an
internal mirror. A Bun-native cooldown keeps bun.lock as the single source of
truth and needs no lockfile regeneration at all.

Point 3 — the publish workflows no longer run the embargo gateway:

- Drops the `sudo sed -i /etc/hosts` unpin hack from both. There is no pin to
  strip now, so the window where the gateway was bypassed is gone entirely.
- Safe because those jobs resolve nothing: `--frozen-lockfile` installs bun.lock
  verbatim, so dropping the gateway does not widen what they can pull.
- check_wix_proxy_steps.py enforces the split bidirectionally: non-publish jobs
  must run the proxy, publish jobs must not. Exemptions are keyed on exact
  filename and printed on success, so the list cannot quietly grow.

Also closes the remaining unprotected registry fetches in the publish path:

- Authenticate via npm trusted publishing (OIDC). Drops NODE_AUTH_TOKEN /
  secrets.NPM_TOKEN from preview-publish; manual-publish never had a credential
  wired at all, so it could not have published regardless.
- Replace `bunx json-bump` with `npm version` and `npm pkg set`. json-bump is
  declared in no manifest, so bunx fetched it at run time — outside bun.lock and
  outside the cooldown, since bunx accepts --minimum-release-age without
  enforcing it (oven-sh/bun#30748).
- Remove `npm install -g npm@latest` from both publish workflows.
- Least-privilege per-job permissions replace the workflow-level blocks; drops
  manual-publish's unused packages: write and pull-requests: read.

Known Bun gaps, documented in bunfig.toml and docs/AGENTS.md rather than left to
be discovered: bunx does not enforce the flag (#30748), `bun update --latest`
skips transitive deps (#25305), and there is no bypass for vulnerability fixes
(#26065) short of minimumReleaseAgeExcludes.

Verified: check_wix_proxy_steps.py reports 12 of 12 non-publish jobs across 11
workflows with both publish jobs confirmed to abstain; 12 of 12 unit tests pass
(4 new, covering both directions of the rule).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dima Ryskin asked for a test rather than a code review of #597: "Can we stack
another PR on top of that and check if installing a fresh-package is rejected? I
used npmjs.com/package/electron-nightly for always 'fresh' versions".

Adds .github/workflows/cooldown-check.yml plus check_cooldown.sh, which probes
the real bunfig.toml setting in a temp project. Three cases, because an exit code
alone does not distinguish a working guardrail from a broken probe:

  control    `bun add lodash` must succeed, so a red result means the cooldown
             fired rather than the environment being broken.
  floating   `bun add electron-nightly` must not land a version inside the
             cooldown — either refused, or resolved to an older one. Asserted on
             the resolved version's publish time from the registry, so silently
             installing a one-day-old version cannot pass.
  exact pin  `bun add electron-nightly@<newest>` must be refused. This is the
             bypass that matters: a PR pinning an exact fresh version.

The cooldown is read out of bunfig.toml rather than hardcoded, so the test cannot
drift from the setting it verifies.

Two constraints shaped the workflow:

- It must NOT run the embargo gateway. Gatewayed, embargo would reject the fresh
  release itself and the run would prove nothing about Bun. check_wix_proxy_steps
  gains a third exemption, and now records why each one exists: the frozenset
  becomes a dict of path -> reason, printed per exemption on success, so an
  exemption cannot be added without stating its justification.
- No third-party actions. The org now sets github_owned_allowed with an empty
  patterns_allowed and requires SHA pinning, so oven-sh/setup-bun is not usable
  here; Bun is installed from a run step and actions/checkout is pinned to
  3d3c42e5 (v7.0.1).

Verified: bash -n clean; bunfig parsing smoke-tested (604800 -> 7 days);
check_wix_proxy_steps reports 12 of 12 gatewayed jobs with all 3 exempt jobs
confirmed to abstain; 13 of 13 unit tests pass (1 new, asserting each exemption
prints its reason).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@guyofeck

Copy link
Copy Markdown
Contributor Author

@dimar this is the test you asked for, and it passed on this PR.

Run: https://github.com/base44/cli/actions/runs/31491517277

Cooldown under test: 604800s (7 days)
Bun: 1.3.14

── control: bun add lodash ──────────────────────────────
PASS  control package installed, so the probe environment works

── floating: bun add electron-nightly ───────────────────────────────
PASS  resolved 45.0.0-nightly.20260803, published 679956s ago (>= cooldown)

── exact pin: bun add electron-nightly@<newest> ─────────────────────
Newest published: 45.0.0-nightly.20260810 (52760s old)
PASS  bun refused the exact fresh pin 45.0.0-nightly.20260810

RESULT: cooldown holds — fresh releases cannot enter the dependency tree

Reading the numbers: Bun skipped the Aug 10 nightly (52,760s ≈ 15 hours old) and resolved back to the Aug 3 one (679,956s ≈ 7.9 days). When pinned explicitly to the fresh version it refused outright — so both the floating resolve and the exact-pin bypass are closed. electron-nightly was your suggestion and it works well for this; it publishes nightly, so there is always something inside the window to reject.

The control case matters for trust in the result: bun add lodash succeeding proves a red run would mean the cooldown fired, not that the probe was broken. And the floating case asserts on the resolved version's publish time from the registry rather than on the exit code, so "silently installed something 15 hours old" cannot pass as green.

The run was ungatewayed on purpose — with embargo in front, the gateway would have rejected the fresh version itself and this would have proven nothing about Bun.


Separate problem this run surfaced. Every other workflow on this PR came back startup_failure:

Workflow Result
Supply-chain Cooldown Check ✅ success
Package Preview Publish ❌ startup_failure
Check Wix Gateway Proxy ❌ startup_failure
Auto PR Description ❌ startup_failure
Claude Code Review ❌ startup_failure

The repo's policy is now github_owned_allowed: true, verified_allowed: false, patterns_allowed: [], plus sha_pinning_required: true. That blocks oven-sh/setup-bun, which lint, test, typecheck, knip and both publish workflows all need — and 15 of our 16 uses: refs are still tag-pinned. So the repo currently cannot run its own CI.

This workflow only ran because it was written to need neither: Bun installs from a run step and actions/checkout is SHA-pinned to 3d3c42e5 (v7.0.1).

Which would you prefer:

  1. Allowlist the actions we need (oven-sh/setup-bun, denoland/setup-deno, and for the agent/publish workflows PostHog/upload-source-maps, peter-evans/repository-dispatch, anthropics/claude-code-action, lycheeverse/lychee-action), or
  2. We drop third-party actions entirely and install toolchains via curl, the way this cooldown check does — more work up front, but then the allowlist never needs touching again.

Either way we'll do the SHA-pinning pass across all workflows; SHA-pinning alone won't help while patterns_allowed is empty.

Base automatically changed from ci/embargo-interim-npm-min-release-age to main August 11, 2026 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant