macOS: code-sign + notarize DisplayXR-Installer-*.pkg (Step 3 of #277)

[dfattal]

Follow-up to #277 Step 3, explicitly deferred as separable.

Today's DisplayXR-Installer-X.Y.Z.pkg (shipping post-#277, #279) is unsigned. Users get Gatekeeper's "DisplayXR cannot be verified, this software needs to be updated" prompt on double-click and need the right-click → Open workaround. sudo installer -pkg from terminal works regardless of signing. This is acceptable for the current developer-facing audience but blocks any end-user-facing macOS distribution.

This issue tracks the work + prerequisites for shipping a clickable, Gatekeeper-clean installer.

Prerequisites (not in repo today)

All of these need to land before the workflow changes below can be made — none of them are code we can write speculatively.

1. Apple Developer Program membership ($99/yr, individual or organizational). Org enrollment recommended (more useful for team-cert sharing) but takes ~1 week of D-U-N-S verification.
2. Two certs from developer.apple.com → Certificates:
   • Developer ID Installer — signs the .pkg itself.
   • Developer ID Application — signs the embedded cube_handle_vk_macos binary inside DisplayXRCube.app (Gatekeeper checks .app content too; unsigned binary inside a signed .pkg still trips the .app launch).
3. App-specific password at appleid.apple.com → Sign-In → App-Specific Passwords. Alternatively, an App Store Connect API key (AuthKey_XXXXXXXXXX.p8 + key id + issuer id) — more robust for CI but more setup.
4. GitHub Actions secrets populated on DisplayXR/displayxr-runtime:
   • MACOS_INSTALLER_CERT_P12 — base64-encoded Developer ID Installer.p12
   • MACOS_INSTALLER_CERT_PASSWORD
   • MACOS_APP_CERT_P12 — base64-encoded Developer ID Application.p12
   • MACOS_APP_CERT_PASSWORD
   • APPLE_ID — Apple ID email (e.g. releases@displayxr.org)
   • APPLE_APP_SPECIFIC_PASSWORD
   • APPLE_TEAM_ID — 10-character team ID (e.g. ABC1234DEF)
5. Forks: secrets aren't exposed to fork PRs. The workflow needs to gracefully fall through to unsigned builds on forks (skip the signing steps; produce a DisplayXR-Installer-...-unsigned.pkg for clarity). Notarized builds only on main push + v* tags from this repo.

Implementation sketch (~40 lines added to build-macos.yml's BuildInstaller job)

- name: Import signing certs
  if: needs.DetectChanges.outputs.docs_only != 'true' && github.event_name != 'pull_request'
  env:
    INSTALLER_CERT_P12: ${{ secrets.MACOS_INSTALLER_CERT_P12 }}
    INSTALLER_CERT_PASSWORD: ${{ secrets.MACOS_INSTALLER_CERT_PASSWORD }}
    APP_CERT_P12: ${{ secrets.MACOS_APP_CERT_P12 }}
    APP_CERT_PASSWORD: ${{ secrets.MACOS_APP_CERT_PASSWORD }}
  run: |
    # Create a temporary keychain so the certs don't pollute the runner default
    KEYCHAIN=$RUNNER_TEMP/build.keychain
    KEYCHAIN_PASSWORD=$(uuidgen)
    security create-keychain -p "$KEYCHAIN_PASSWORD" "$KEYCHAIN"
    security default-keychain -s "$KEYCHAIN"
    security unlock-keychain -p "$KEYCHAIN_PASSWORD" "$KEYCHAIN"
    security set-keychain-settings -lut 21600 "$KEYCHAIN"
    # Decode + import both certs
    echo "$INSTALLER_CERT_P12" | base64 -d > "$RUNNER_TEMP/installer.p12"
    echo "$APP_CERT_P12" | base64 -d > "$RUNNER_TEMP/app.p12"
    security import "$RUNNER_TEMP/installer.p12" -k "$KEYCHAIN" -P "$INSTALLER_CERT_PASSWORD" -T /usr/bin/productsign
    security import "$RUNNER_TEMP/app.p12" -k "$KEYCHAIN" -P "$APP_CERT_PASSWORD" -T /usr/bin/codesign
    security set-key-partition-list -S apple-tool:,apple:,codesign:,productsign: -s -k "$KEYCHAIN_PASSWORD" "$KEYCHAIN" >/dev/null
    rm -f "$RUNNER_TEMP/installer.p12" "$RUNNER_TEMP/app.p12"

- name: Sign embedded test app binary
  if: needs.DetectChanges.outputs.docs_only != 'true' && github.event_name != 'pull_request'
  env:
    APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
  run: |
    # Sign every Mach-O inside the .app that Gatekeeper inspects.
    # Order matters: deep-most artifacts first, .app shell last.
    find _package/DisplayXR-macOS/bin -type f -perm +111 | while read bin; do
      codesign --force --options runtime --timestamp \
        --sign "Developer ID Application: <Org Name> ($APPLE_TEAM_ID)" "$bin"
    done

# (BuildInstaller calls scripts/build_macos.sh --installer here, producing
# _package/DisplayXR-Installer-${VERSION}.pkg unsigned.)

- name: Sign + notarize installer
  if: needs.DetectChanges.outputs.docs_only != 'true' && github.event_name != 'pull_request'
  env:
    APPLE_ID: ${{ secrets.APPLE_ID }}
    APPLE_APP_SPECIFIC_PASSWORD: ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
    APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
  run: |
    UNSIGNED=$(ls _package/DisplayXR-Installer-*.pkg | head -1)
    SIGNED="${UNSIGNED%.pkg}-signed.pkg"
    productsign --sign "Developer ID Installer: <Org Name> ($APPLE_TEAM_ID)" \
      "$UNSIGNED" "$SIGNED"
    # Submit to Apple's notary service (blocks until done; typically 1-3 min)
    xcrun notarytool submit "$SIGNED" \
      --apple-id "$APPLE_ID" \
      --password "$APPLE_APP_SPECIFIC_PASSWORD" \
      --team-id "$APPLE_TEAM_ID" \
      --wait
    # Staple the notarization ticket so offline machines can verify
    xcrun stapler staple "$SIGNED"
    # Replace the unsigned .pkg with the signed+notarized one
    mv "$SIGNED" "$UNSIGNED"
    # Verify spctl accepts it
    spctl --assess --type install --verbose "$UNSIGNED"

Acceptance

• Apple Developer Program enrollment complete
• All seven GH Actions secrets populated
• CI run from main produces a notarized + stapled .pkg
• spctl --assess --type install --verbose DisplayXR-Installer-*.pkg returns accepted with source=Notarized Developer ID
• Double-clicking the .pkg on a clean macOS opens the installer with no Gatekeeper warning
• Fork PRs continue to produce unsigned .pkgs and don't fail the workflow
• CLAUDE.md + docs/getting-started/building.md updated to drop the right-click → Open workaround

When to schedule

Not urgent for the current developer-facing audience. Worth doing when one of:

• DisplayXR Shell on macOS ships (currently deferred, CLAUDE.md M6)
• First end-user-facing demo .app ships from this repo
• A vendor partner asks for it

Until then, the unsigned .pkg + right-click → Open workaround is documented (post this issue) and acceptable.

References

• .github/workflows/build-macos.yml — the workflow to extend
• installer/macos/build_installer.sh — produces the unsigned .pkg we'd then sign
• Apple notarization docs