diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9cb9e6e..1b52926 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -55,6 +55,9 @@ jobs: - name: Test public release metadata and documentation run: npm run smoke:public-release + - name: Test release metadata contract + run: npm run smoke:release-metadata + - name: Test canonical release identity run: npm run smoke:release-identity @@ -76,6 +79,9 @@ jobs: - name: Install backend smoke dependencies run: python -m pip install fastapi pydantic python-multipart requests + - name: Test deterministic model manager workflow + run: npm run smoke:model-manager + - name: Lint frontend run: npm run lint diff --git a/.github/workflows/release-unsigned.yml b/.github/workflows/release-unsigned.yml index db14f23..f710cea 100644 --- a/.github/workflows/release-unsigned.yml +++ b/.github/workflows/release-unsigned.yml @@ -277,6 +277,23 @@ jobs: hdiutil detach "$MOUNT_POINT" -force >/dev/null echo "gatekeeper_assessment_exit=$SPCTL_STATUS (diagnostic; unsigned public alpha is not expected to be Apple-accepted)" + - name: Run bounded packaged functional proof from exact DMG + run: | + set -euo pipefail + DMG="dist/public-release/$(node -p "require('./dist/public-release/release-manifest.json').artifact.filename")" + MOUNT_POINT="$RUNNER_TEMP/scriptcut-functional-mount" + mkdir -p "$MOUNT_POINT" + cleanup() { + hdiutil detach "$MOUNT_POINT" -force >/dev/null 2>&1 || true + } + trap cleanup EXIT + hdiutil attach "$DMG" -readonly -nobrowse -mountpoint "$MOUNT_POINT" >/dev/null + APP_PATH="$MOUNT_POINT/ScriptCut.app" + test -d "$APP_PATH" + echo "Exact mounted public app: $APP_PATH" + node scripts/smoke-packaged-release-identity.js --app "$APP_PATH" + node scripts/check-packaged-electron-backend.js --app "$APP_PATH" + publish: name: Publish verified GitHub prerelease if: ${{ inputs.publish == true }} diff --git a/docs/CI.md b/docs/CI.md index 2970084..c81dec9 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -1,17 +1,27 @@ # Continuous Integration -The recommended GitHub Actions checks for ScriptCut are: +Pull requests and pushes to `main` run the `checks` job in +`.github/workflows/ci.yml`. The source-level release and product checks include: ```bash -npm install --package-lock=false --no-audit --no-fund +npm ci --no-audit --no-fund npm ci --prefix frontend python -m pip install fastapi pydantic python-multipart requests npm run lint npm run build --prefix frontend +npm run smoke:model-manager npm run smoke:backend python -m compileall -q backend ``` -The backend smoke checks intentionally use minimal Python dependencies so CI does not need to install the full transcription and ML stack for every pull request. +The workflow also runs the release identity, metadata, public fixture, +workflow-structure, runtime-contract, renderer-policy, brand, and frontend +workflow smokes. The model-manager smoke uses a deterministic loopback fixture; +it does not download a real model. Backend smoke checks intentionally use +minimal Python dependencies so CI does not need to install the full +transcription and ML stack for every pull request. -When repository automation has permission to create workflow files, add a GitHub Actions workflow that runs the commands above on pushes to `main` and on pull requests. +The manually dispatched `release_candidate` path builds the native arm64 +candidate. Public dry-runs and publication are governed separately by +[Release QA](./RELEASE_QA.md) and +`.github/workflows/release-unsigned.yml`. diff --git a/docs/DESKTOP_QA.md b/docs/DESKTOP_QA.md index eb47c60..e182f32 100644 --- a/docs/DESKTOP_QA.md +++ b/docs/DESKTOP_QA.md @@ -1,6 +1,12 @@ # Desktop QA Checklist -Use this checklist before publishing a desktop build or handing a build to creators. +Use this checklist when manual physical creator qualification is required for a +release candidate or public alpha. It is required for changes to installer or +DMG layout, signing/trust behavior, runtime packaging, backend startup, +renderer transport/CSP, model first-use flow, video open/preview, +transcription, or core export. Release-system-only changes that do not alter +the public artifact or creator runtime may mark it not required. See +[Release QA](./RELEASE_QA.md) for the full gate lifecycle. ## Automated Gate diff --git a/docs/RELEASE.md b/docs/RELEASE.md index cadc742..8201289 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -2,6 +2,9 @@ This guide is for preparing a desktop release from the repository. +The durable gate ownership and lifecycle matrix is documented in +[Release QA](./RELEASE_QA.md). + ## Current Release Status The release system uses an explicit identity contract while preserving the controlled public ad-hoc-signed macOS arm64 distribution path. This guide describes how qualifying public alphas are produced; it does not assert that every historical alpha already has the self-contained runtime. Inspect each release's notes and manifest. The workflow creates no tag or GitHub Release unless explicitly dispatched with publication gates. Apple Developer membership, Developer ID signing, and notarization are optional future enhancements, not prerequisites for the public ad-hoc alpha path. Source development is still supported with: @@ -51,6 +54,11 @@ When packaging changes are included, run: npm run qa:desktop:package ``` +These local and developer commands produce QA builds only. They are not public +release artifacts. The authoritative candidate command is +`npm run release:rc:arm64`; the authoritative public release is the GitHub +Actions **Release unsigned alpha** workflow. + Check ad-hoc candidate readiness: ```bash @@ -110,7 +118,9 @@ The dedicated `.github/workflows/release-unsigned.yml` workflow is `workflow_dis Required inputs are `release_tag`, `publish`, `real_model`, and `confirmation`. The tag must be `v-alpha.` and must be greater than every existing alpha tag. `publish=false` is the safe dry-run mode: it may create attestations and a workflow artifact whose name includes `dry-run`, but it cannot create a tag, release, commit, or mutate `main`. Publication additionally requires `publish=true`, `real_model=true`, `confirmation=PUBLISH_UNSIGNED_ALPHA`, the workflow ref to be `main`, and a current `origin/main` equal to the dispatched commit. The publish job has contents write permission only, downloads the already verified artifact, creates a GitHub prerelease with `--prerelease --latest=false`, and verifies the exact tag, asset set, and digest after creation. It never rebuilds during publication. -The post-merge closure procedure is a `publish=false` dry-run from current `main`. It is required before any separately authorized public publication and is not performed by this implementation PR. +The closure procedure is a `publish=false` dry-run from current `main`. Run it +before any separately authorized public publication; it creates no tag or +GitHub Release. Title: diff --git a/docs/RELEASE_QA.md b/docs/RELEASE_QA.md new file mode 100644 index 0000000..b55341b --- /dev/null +++ b/docs/RELEASE_QA.md @@ -0,0 +1,113 @@ +# Release QA Gate Lifecycle + +This document is the durable release QA contract for ScriptCut. It separates +source correctness, packaged candidate behavior, public artifact verification, +manual product qualification, publication, and post-publication checks. + +The identity contract is intentionally unchanged: + +- `productVersion` is `0.1.0` and the implemented public channel is `alpha`. +- The installed app uses `productVersion`; the public release uses + `releaseTag` (`v0.1.0-alpha.`). +- `npm run release:rc:arm64` is the authoritative native candidate command. +- GitHub Actions → **Release unsigned alpha** is the authoritative public + release path. +- A developer build, candidate, dry-run, qualified public artifact, and + published release are different states. + +## Gate matrix + +| Gate | Owner stage | Runner | What it proves | Required | Conditional | Network / model download | Can mutate GitHub | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Frontend lint and build | SOURCE / PR | `ubuntu-latest` | Source-level frontend lint and compilation/build correctness | Yes for PR CI | No | CI dependency install uses the network; no model download | No | +| Backend smoke and compile | SOURCE / PR | `ubuntu-latest` | Backend source behavior and Python syntax without the packaged runtime | Yes for PR CI | No | CI dependency install uses the network; no model download | No | +| Release identity, metadata, public fixture, workflow, runtime contract, and renderer policy smokes | SOURCE / PR | `ubuntu-latest` | Stable release identity, metadata, public documentation fixture, workflow structure, runtime contract, and renderer policy | Yes for PR CI | No | No model download | No | +| Deterministic model-manager smoke | SOURCE / PR | `ubuntu-latest` | Managed model storage, resume, cancellation, integrity, redirect policy, and path safety against a local fixture server | Yes for PR CI | No | Loopback fixture only; no real model download | No | +| Native candidate build | CANDIDATE | `macos-14` arm64 | The requested commit produces the self-contained native macOS arm64 candidate | Yes before public qualification | No | Package prerequisites use the network; real model is optional for a candidate invocation | No | +| Packaged candidate gates | CANDIDATE | `macos-14` arm64 | Packaged runtime, bundled backend, Electron-like startup, renderer transport/CSP, optional-capability isolation, FFmpeg, transcription, icon, DMG, metadata, identity, and signing readiness | Yes for a candidate | No | No model download unless the candidate is invoked with `--real-model` | No | +| Candidate real-model gate | CANDIDATE | `macos-14` arm64 | First acquisition, verified model, real transcription, offline reuse, repair, and reacquisition when explicitly requested | No for every candidate | Yes: `--real-model` or an implementation change to real-model orchestration | Downloads the real model when enabled; hosted execution is CPU-only | No | +| Public build, stage, and attestation | PUBLIC DRY-RUN | `macos-14` arm64 | One built candidate is transformed into the exact public bundle, with checksums, manifest, DMG and manifest attestations, and no rebuild after staging | Yes for a public dry-run or publication | `workflow_dispatch`; `publish=false` is the non-publishing mode | Network required for actions, GitHub attestations, and `real_model=true` validation | Attestation and workflow-artifact records only; `publish=false` cannot create a tag, release, commit, or main mutation | +| Transferred public bundle checksum, manifest, DMG, and attestation verification | CLEAN ARTIFACT | `macos-14` arm64 | The exact transferred public bundle still has the expected shape, bytes, provenance, and native DMG validity | Yes for a public dry-run or publication | No once the public workflow is run | Network required for GitHub attestation verification; no model download | No | +| Mounted exact-DMG packaged identity proof | CLEAN ARTIFACT | `macos-14` arm64 | `ScriptCut.app` inside the exact mounted public DMG reports the canonical packaged identity | Yes for a public dry-run or publication | No once the public workflow is run | No model download | No | +| Mounted exact-DMG bundled backend startup proof | CLEAN ARTIFACT | `macos-14` arm64 | The exact mounted app starts its bundled local backend and passes the bounded authenticated health/diagnostics checks under a Finder-like environment | Yes for a public dry-run or publication | No once the public workflow is run | No model download; system/developer Python is not accepted | No | +| Gatekeeper assessment diagnostic | CLEAN ARTIFACT | `macos-14` arm64 | The trust state is reported honestly for the unsigned alpha without disabling Gatekeeper or changing xattrs | Yes for a public dry-run or publication | No once the public workflow is run | No model download | No | +| Physical MPS qualification | MANUAL QUALIFICATION | Physical Apple Silicon Mac | Actual MPS behavior for changes touching Torch/MPS selection, Whisper timing, GPU execution, or native GPU dependencies | No for unrelated changes | Yes when an MPS-sensitive surface changes | May use the real model; use the release-specific test procedure | No | +| Physical creator qualification | MANUAL QUALIFICATION | Physical Mac with the installed candidate/public DMG | Installer/layout, trust behavior, first-use model flow, video open/preview, transcription, renderer transport/CSP, backend startup, and core export behavior | No for release-system-only changes | Yes for the current alpha when those creator-facing surfaces change | May require first-use model download and creator media | No | +| Publication gate and exact prerelease creation | PUBLISH | `macos-14` | Current-main binding, explicit confirmation, `real_model=true`, monotonic unused tag, exact verified artifact, no rebuild, and prerelease semantics | No for a dry-run | Yes only when `publish=true` is explicitly dispatched | GitHub API/tag/release access and real-model proof are required | Yes: creates the GitHub prerelease and tag only after all gates pass | +| Published tag, prerelease, asset-set, and digest verification | POST-PUBLISH | `macos-14` | The published GitHub prerelease points at the dispatched commit and contains the exact expected assets and digest | Yes after publication | No when publication occurred | GitHub API and tag access; no model download | No | + +The clean artifact stage is intentionally bounded. It does not repeat the +candidate's full transcription, optional-capability, icon-generation, runtime +preparation, or physical MPS matrix. Its functional question is only whether +the exact verified public DMG exposes the packaged identity and starts its +bundled local backend on a fresh native runner. + +## Source and candidate ownership + +Source / PR gates are cheap, deterministic checks of source and contract +regressions. A candidate is expected to originate from a commit that passed +source CI; candidate gates prove packaged behavior and do not replace source +CI. + +The native candidate command is: + +```bash +npm run release:rc:arm64 +``` + +Use `npm run release:rc:arm64 -- --real-model` only when the extended model +gate is required. Hosted runners may prove the CPU real-model baseline and a +deterministic or simulated MPS compatibility branch; they must not claim +physical MPS qualification. + +## Manual qualification triggers + +For the current alpha lifecycle, manual physical creator qualification is +required when a release changes installer or DMG layout, signing/trust +behavior, runtime packaging, backend startup, renderer transport/CSP, model +first-use flow, video open/preview, transcription, or the core export path. +Release-system-only changes that do not alter the public artifact or creator +runtime may mark creator qualification as not required. + +Physical MPS validation is required only when the change touches Torch/MPS +device behavior, Whisper MPS timing compatibility, GPU execution selection, +MPS-specific transcription code, or native GPU dependency/runtime behavior. +It is not a standing gate for unrelated releases, and existing +`--require-mps` semantics remain unchanged. + +## Command classification + +These commands are useful developer or QA paths, but they do not create public +release artifacts: + +```text +npm run dist +npm run dist:mac +npm run dist:mac:arm64 +npm run dist:mac:arm64:self-contained +npm run qa:desktop:package +``` + +The compatibility aliases remain available: + +- `release:alpha` invokes the same candidate orchestrator as + `release:rc:arm64`. +- `release:trust` and `release:trust:candidate` invoke the same candidate + signing-readiness check. +- `release:trust:signed` is the separate credentialed future signed-mode + check; it is not the public unsigned-alpha publisher. + +There is no public npm publisher. Publication remains exclusively owned by +`.github/workflows/release-unsigned.yml`, which builds once, stages exact +candidate bytes, attests them, verifies them on a clean native runner, and +publishes only the already verified artifact. + +## Trust and mutation invariants + +The public path remains current-main-only and requires +`confirmation=PUBLISH_UNSIGNED_ALPHA` plus `real_model=true`. It verifies the +source commit, SHA-256, GitHub artifact provenance, native DMG, and ad-hoc +signing state. Developer ID signing, notarization, and Hardened Runtime remain +false for the current public path. Gatekeeper output is diagnostic only; no +xattr mutation or Gatekeeper disabling is permitted, and secrets must not enter +manifests or logs. diff --git a/scripts/check-release-workflow.js b/scripts/check-release-workflow.js index 774004e..5d94763 100644 --- a/scripts/check-release-workflow.js +++ b/scripts/check-release-workflow.js @@ -35,6 +35,36 @@ function stepBlock(text, stepName) { return text.slice(after, next < 0 ? text.length : after + next); } +function needsList(block, jobName) { + const match = block.match(/^\s+needs:\s*(.+)$/m); + assert(match, `${jobName} job dependency declaration is missing`); + return match[1] + .replace(/[\[\]]/g, '') + .split(',') + .map((value) => value.trim()) + .filter(Boolean); +} + +function validateSourceCiGates(text) { + const checks = jobBlock(text, 'checks'); + const requiredCommands = [ + 'npm run smoke:release-identity', + 'npm run smoke:release-metadata', + 'npm run smoke:public-release', + 'npm run smoke:release-workflow', + 'npm run smoke:runtime-contract', + 'npm run smoke:renderer-policy', + 'npm run smoke:model-manager', + 'npm run lint', + 'npm run build --prefix frontend', + 'npm run smoke:backend', + 'python -m compileall -q backend', + ]; + for (const command of requiredCommands) { + assert(checks.includes(command), `normal PR CI is missing ${command}`); + } +} + function validateOpenFileGuard(text, label, stepName) { const build = stepBlock(text, stepName); const guardError = (suffix) => [label, suffix].join(' '); @@ -89,6 +119,9 @@ function validateWorkflowText(text, { candidateWorkflowText = fs.readFileSync(ci const build = jobBlock(text, 'build'); const clean = jobBlock(text, 'clean-runner-verify'); const publish = jobBlock(text, 'publish'); + assert(needsList(clean, 'clean-runner-verify').length === 1 && needsList(clean, 'clean-runner-verify')[0] === 'build', 'clean runner must depend on build'); + const publishNeeds = needsList(publish, 'publish'); + assert(publishNeeds.includes('build') && publishNeeds.includes('clean-runner-verify'), 'publish job must depend on build and clean-runner-verify'); assert(/runs-on:\s+macos-14/.test(build), 'build runner must be macos-14'); assert(/contents:\s+read/.test(build), 'build contents permission must be read'); assert(/id-token:\s+write/.test(build), 'build id-token permission must be write'); @@ -107,6 +140,7 @@ function validateWorkflowText(text, { candidateWorkflowText = fs.readFileSync(ci validateIconBuildPrerequisites(candidateWorkflowText, 'ci.yml', 'Install release build prerequisites'); assert(/runs-on:\s+macos-14/.test(clean), 'clean verification runner must be macos-14'); + assert(/uname -m[\s\S]*?arm64/.test(clean) && /node -p process\.arch[\s\S]*?arm64/.test(clean), 'clean runner must verify native macOS arm64'); assert(/actions\/download-artifact@v4/.test(clean), 'clean runner must download the build artifact'); assert(/gh attestation verify/.test(clean), 'clean runner attestation verification is missing'); assert(/--repo/.test(clean) && /--signer-workflow/.test(clean) && /--source-digest/.test(clean), 'clean runner does not constrain attestation identity enough'); @@ -114,6 +148,10 @@ function validateWorkflowText(text, { candidateWorkflowText = fs.readFileSync(ci assert((clean.match(/--bundle/g) || []).length >= 2, 'clean runner must verify both downloaded local attestation bundles'); assert(/DMG_BASENAME/.test(clean) && /release-manifest\.sigstore\.json/.test(clean), 'local DMG and manifest bundle paths are not both verified'); assert(/hdiutil verify/.test(clean) || /hdiutil attach/.test(clean), 'clean runner DMG verification is missing'); + assert(/test -d "\$APP_PATH"/.test(clean), 'clean runner must locate ScriptCut.app inside the mounted DMG'); + assert(/smoke-packaged-release-identity\.js --app "\$APP_PATH"/.test(clean), 'clean runner packaged identity proof is missing'); + assert(/check-packaged-electron-backend\.js --app "\$APP_PATH"/.test(clean), 'clean runner packaged backend startup proof is missing'); + assert(!/--real-model|smoke-packaged-transcription|smoke-packaged-optional-capabilities|runtime:prepare|release:rc:arm64/.test(clean), 'clean runner must not repeat candidate or real-model validation'); assert(/spctl --assess/.test(clean), 'clean runner Gatekeeper diagnostic is missing'); assert(!/spctl --master-disable|xattr\s+-dr|sudo\s+spctl/.test(clean), 'Gatekeeper bypass command found'); assert(!/id-token:\s+write|attestations:\s+write/.test(clean), 'clean runner has build-only attestation permissions'); @@ -143,6 +181,7 @@ function main() { validateWorkflowText(fs.readFileSync(workflowPath, 'utf8'), { candidateWorkflowText: fs.readFileSync(ciWorkflowPath, 'utf8'), }); + validateSourceCiGates(fs.readFileSync(ciWorkflowPath, 'utf8')); validateHostedCandidateInvocation(fs.readFileSync(ciWorkflowPath, 'utf8')); validateReleaseArgumentSeparation(fs.readFileSync(releaseAlphaPath, 'utf8')); const releaseConfig = fs.readFileSync(releaseConfigPath, 'utf8'); diff --git a/scripts/smoke-release-metadata.js b/scripts/smoke-release-metadata.js index 9c7a581..2638423 100644 --- a/scripts/smoke-release-metadata.js +++ b/scripts/smoke-release-metadata.js @@ -28,6 +28,55 @@ function walkValues(value, visit) { else if (value && typeof value === 'object') Object.values(value).forEach((entry) => walkValues(entry, visit)); } +function createSourceFixture() { + const fixtureRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'scriptcut-release-metadata-fixture-')); + const releaseDir = path.join(fixtureRoot, 'release-candidate'); + fs.mkdirSync(releaseDir, { recursive: true }); + + const productVersion = readProductVersion(); + const filename = formatCandidateArtifactFilename(productVersion, 'arm64'); + const artifact = Buffer.from('ScriptCut source release metadata fixture\n'); + const artifactSha256 = crypto.createHash('sha256').update(artifact).digest('hex'); + const manifest = { + schema: 'scriptcut.release.v1', + version: productVersion, + platform: 'darwin', + architecture: 'arm64', + commit: 'f'.repeat(40), + tagCandidate: null, + tagExists: false, + signed: false, + notarized: false, + codeSignature: { type: 'ad-hoc', structurallyValid: true, hardenedRuntime: false }, + model: { embedded: false, manifestSha256: '1'.repeat(64) }, + runtime: { + mode: 'packaged-bundled', + pythonSource: 'bundled', + target: { platform: 'darwin', arch: 'arm64' }, + manifestSha256: '2'.repeat(64), + }, + coreInventorySha256: '3'.repeat(64), + ffmpeg: { manifestSha256: '4'.repeat(64) }, + artifact: { filename, bytes: artifact.length, sha256: artifactSha256 }, + }; + const notes = [ + 'Creators do not need to install Python, run pip, download FFmpeg, configure PATH, or create a virtual environment.', + 'The first transcription downloads the model; later use works without model-network access.', + 'This is an internal release candidate and is not for public distribution.', + 'The artifact has an ad-hoc structural signature and is not signed with Apple Developer ID.', + ].join('\n'); + + fs.writeFileSync(path.join(releaseDir, filename), artifact); + fs.writeFileSync(path.join(releaseDir, 'release-manifest.json'), `${JSON.stringify(manifest, null, 2)}\n`); + fs.writeFileSync(path.join(releaseDir, 'SHA256SUMS.txt'), `${artifactSha256} ${filename}\n`); + fs.writeFileSync(path.join(releaseDir, 'RELEASE_NOTES.md'), `${notes}\n`); + + return { + releaseDir, + cleanup: () => fs.rmSync(fixtureRoot, { recursive: true, force: true }), + }; +} + async function testStreamingChecksum() { const fixtureRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'scriptcut-release-hash-')); try { @@ -47,8 +96,7 @@ async function testStreamingChecksum() { } } -async function main() { - const releaseDir = path.resolve(optionValue('--dir') || path.join(root, 'dist', 'release-candidate')); +async function validateReleaseDirectory(releaseDir) { const manifestPath = path.join(releaseDir, 'release-manifest.json'); const checksumPath = path.join(releaseDir, 'SHA256SUMS.txt'); const notesPath = path.join(releaseDir, 'RELEASE_NOTES.md'); @@ -97,6 +145,17 @@ async function main() { console.log('Release metadata schema, provenance, path safety, notes, artifact checksum, and streaming checksum tests passed.'); } +async function main() { + const requestedDir = optionValue('--dir'); + const sourceFixture = requestedDir ? null : createSourceFixture(); + const releaseDir = path.resolve(requestedDir || sourceFixture.releaseDir); + try { + await validateReleaseDirectory(releaseDir); + } finally { + sourceFixture?.cleanup(); + } +} + main().catch((error) => { console.error(error instanceof Error ? error.message : String(error)); process.exit(1);