Enable independent Core and DSP owner updates

.github/workflows/checks.yml

@@ -32,6 +32,12 @@ jobs:
       - run: npm run check
       - run: npm test
       - run: npm run build -- "$RUNNER_TEMP/core-candidate"
+      - name: Install browser for Updates checks
+        working-directory: dashboard
+        run: npx playwright install --with-deps chromium
+      - name: Verify owner Updates workflow in the browser
+        working-directory: dashboard
+        run: npx playwright test --config playwright.updates.config.cjs
       - run: tar -C "$RUNNER_TEMP/platform-packages" -czf "$RUNNER_TEMP/platform-packages.tar.gz" .
       - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:

DEVELOPMENT.md

@@ -73,3 +73,6 @@ Keep multiline PR bodies in a file and pass `--body-file`.
 
 The manual release workflow is a separate operation; never dispatch it as part of
 ordinary development, merging, testing or retrying CI. See `RELEASES.md`.
+
+Owner update controls, synthetic browser checks and the native DSP release test
+are documented in [core/updates/README.md](core/updates/README.md).

README.md

@@ -11,8 +11,10 @@ Every DSP retains its own installed code and private state.
 - [Directory map](AGENTS.md)
 - [Local development](DEVELOPMENT.md)
 - [PR and release workflow](RELEASES.md)
+- [Independent update operations](core/updates/README.md)
 - [Architecture and storage](docs/architecture.md)
 - [Security policy](SECURITY.md)
 
-This source is being prepared locally for repository creation. No repository,
-release version, permanent preview deployment or production cutover is implied.
+Core and DSP use separate public repositories and release versions. Publishing a
+GitHub release makes it available; installing it is a separate owner action. See
+the update operations guide for initial deployment and recovery prerequisites.

RELEASES.md

@@ -15,28 +15,26 @@ Updating Core never rewrites an installed DSP's runtime, plugins or SDK copies.
 6. Build and verify immutable artifacts from that main commit, collect a readable
    changelog, and publish the selected product's release. Publishing does not install.
 
-The future owner Updates area has independent Core and DSP pages. Core shows its
-changelog and an Update Core action. Shared dashboard/API features should first be
-validated in a separate Core preview connected only to the permanent Dev DSP.
+The owner Updates area has independent **Core** and **DSPs** tabs, release
+history and changelogs. **Update Core** changes the shared dashboard, API and
+services; it leaves installed DSP runtimes, plugins and SDK copies in place.
+Validate shared changes in an isolated Core preview before installing them.
+There is no automatic Core preview or automatic production deployment.
 
-For DSP releases, Update Dev installs only the permanent testing DSP. Successful
-installation and health checks enable Rollout Update. A newer release before
-rollout resets the required Dev test. Rollout processes DSPs one at a time, checks
-each DSP and pauses on failure. A rollout already started stays pinned to its exact
-artifact even if another release is published. Retain previous code and a compatible
-state snapshot for rollback; reverting code alone cannot undo a database migration.
+**Update Dev** installs the latest DSP release only on the configured permanent
+Dev DSP. A successful installation and live health check enable **Rollout Update**.
+The owner tests Dev and chooses when to start rollout. A newer published release
+before rollout requires another Dev update. An active rollout stays pinned to its
+selected artifact, updates one DSP at a time and pauses on failure. A completed
+rollout sets the default version for new DSPs; a Dev candidate never becomes that
+default. DSPs created during a rollout join its remaining queue.
 
-Local foundations are implemented in `core/updates/local-releases.js`,
-`host/releases/runtime.js` and the package catalog's per-DSP approvals. These are
-internal lifecycle ports, not public HTTP installation endpoints. Activation hooks
-must drain processes, snapshot private state, start selected code, verify health
-and restore on failure. Recover interrupted operations explicitly before continuing.
-The persistent state directory is private and is never included in source exports.
-
-The GitHub feed, permanent Dev DSP deployment, separate Core preview, privileged
-activation hooks and owner Updates UI are subsequent work. The development builder
-still produces development candidates. Publication uses the separate guarded
-workflow below; no release version or production baseline is assigned by setup.
+Updates require a verified initial split deployment, a permanent Dev DSP and the
+separate update worker. The first published `0.0.1` Core predates these controls;
+it cannot install this feature by itself. See [update operations](core/updates/README.md)
+for configuration, adoption prerequisites, lifecycle/recovery and test commands.
+The updater never treats a source checkout or development build as a published
+release. Publishing and installation remain separate owner decisions.
 
 Legacy `core/installations/RELEASES.md` documents old native/OCI recovery formats.
 It does not authorize or describe the new release workflow.
@@ -68,5 +66,5 @@ installed bytes is rejected; bump that component in a reviewed PR first.
 The release job has GitHub publication permissions only. There are no production
 SSH credentials, service restarts, deployment hooks or DSP activation steps.
 GitHub publication and installed-version verification are separate operations.
-The first real publication is still pending an owner-selected version; do not
-claim that upload/attestation publication has been exercised by the unit tests.
+Core and DSP `0.0.1` have been published and their GitHub assets and attestations
+verified. That publication did not install them on the platform.

bin/dispatch-updates

@@ -0,0 +1,41 @@
+#!/usr/bin/env -S node --no-warnings
+'use strict';
+process.umask(0o077);
+async function main() {
+  const [action, ...args] = process.argv.slice(2);
+  const { loadWorkerPaths, configure, adopt } = require('../host/releases/setup');
+  const paths = loadWorkerPaths();
+  if (action === 'worker' && !args.length) {
+    const entry = require('../host/releases/setup').workerEntrypoint(paths, __filename);
+    if (entry) {
+      const child = require('node:child_process').spawn(process.execPath, ['--no-warnings', entry, 'worker'], { stdio: 'inherit', env: process.env });
+      process.once('SIGTERM', () => child.kill('SIGTERM')); process.once('SIGINT', () => child.kill('SIGINT'));
+      await new Promise((resolve, reject) => { child.once('error', reject); child.once('exit', code => code === 0 ? resolve() : reject(new Error('release_worker_failed'))); });
+      return { status: 'stopped' };
+    }
+    const app = await require('../core/updates/worker').startWorker(paths);
+    const startedDigest = app.worker.releases.state().active.core;
+    const monitor = setInterval(() => {
+      const state = app.worker.releases.state();
+      if (!state.operation && state.active.core !== startedDigest) close();
+    }, 1000);
+    const close = () => { clearInterval(monitor); return app.close().catch(() => { process.exitCode = 1; }); };
+    process.once('SIGINT', close); process.once('SIGTERM', close);
+    return { status: 'ready' };
+  }
+  if (action === 'configure' && args.length === 2) return configure(paths, args[0], Number(args[1]));
+  if (action === 'adopt' && args.length === 1) return adopt(paths, args[0]);
+  if (action === 'recover' && args.length === 1) {
+    const { authorizeOwner } = require('../core/updates/worker'); authorizeOwner(paths, args[0]);
+    const { UpdateCommands } = require('../core/updates/commands');
+    const { rootFor } = require('../core/updates/configuration');
+    const job = new UpdateCommands(rootFor(paths)).request(args[0], { action: 'recover', product: 'core',
+      digest: null, idempotencyKey: `recovery:${require('node:crypto').randomUUID()}` });
+    return { status: 'queued', id: job.id };
+  }
+  throw new Error('release_command_invalid');
+}
+main().then(value => process.stdout.write(JSON.stringify({ ok: true, ...value }) + '\n')).catch(error => {
+  process.stderr.write(JSON.stringify({ ok: false, status: /^release_[a-z_]+$/.test(error.message) ? error.message : 'release_operation_failed' }) + '\n');
+  process.exitCode = 1;
+});

core/accounts/src/directory-lifecycle.js

@@ -48,6 +48,8 @@ function createDirectoryLifecycle({ store, clock = Date.now }) {
     if (!['suspend', 'resume', 'restart', 'decommission', 'restore_dsp'].includes(action)) fail();
     return store.transaction(() => {
       const control = store.installationControl(organizationId), organization = store.organization(organizationId);
+      if (action === 'decommission' && control?.runtimeKey === store.permanentDevId) fail('directory_dev_protected');
+      if (store.releaseBlocked?.(organizationId)) fail('release_busy');
       if (store.directoryDeletion?.get(organizationId)) fail('installation_operation_not_allowed');
       if (!control || store.installationBackend(organizationId) !== BACKEND) fail();
       const prior = db.prepare('SELECT * FROM directory_lifecycle_requests WHERE organization_id=? AND idempotency_key=?').get(organizationId, requestId);

core/accounts/src/plugins.js

@@ -36,6 +36,7 @@ function createPluginService({ store, access, invoke, settingsPort = null, insta
   function context(session) {
     const selected = access.requireDspOwner(session);
     const installation = store.installationControl(selected.organization.id);
+    if (store.releaseBlocked?.(selected.organization.id)) fail('release_busy');
     if (selected.organization.status !== 'active' || installation?.status !== 'ready'
         || !backends.includes(store.installationBackend(installation.organizationId)) || store.activeLifecycleJob(selected.organization.id)
         || store.db.prepare("SELECT 1 FROM directory_lifecycle_requests WHERE organization_id=? AND status IN ('queued','running')").get(selected.organization.id)) fail('installation_not_ready');
@@ -89,7 +90,7 @@ function createPluginService({ store, access, invoke, settingsPort = null, insta
   }
   function ready(organizationId, runtimeKey = null) {
     const installation = store.installationControl(organizationId);
-    return installation?.status === 'ready' && backends.includes(store.installationBackend(installation.organizationId))
+    return !store.releaseBlocked?.(organizationId) && installation?.status === 'ready' && backends.includes(store.installationBackend(installation.organizationId))
       && (!runtimeKey || installation.runtimeKey === runtimeKey)
       && store.organization(organizationId)?.status === 'active' && !store.activeLifecycleJob(organizationId)
       && !store.db.prepare('SELECT 1 FROM dsp_removals WHERE organization_id=?').get(organizationId)

core/agents/tests/execution.test.js

@@ -7,9 +7,10 @@ const path = require('node:path');
 const { DatabaseSync } = require('node:sqlite');
 const { DirectoryExecution } = require('../../../host/controller/execution');
 const { saveStatus } = require('../../../shared/published/status');
-const { success } = require('../../../shared/contracts/src/result');
+const { success, failure } = require('../../../shared/contracts/src/result');
 
 function fixture(t) {
+  require('../../../shared/plugin-sdk/catalog').configureCatalog(() => [require('../../../tests/fixtures/paycom-plugin.json')]);
   const root = fs.mkdtempSync(path.join(os.tmpdir(), 'core-execution-'));
   const paths = { local: path.join(root, 'local') };
   for (const name of ['local', 'local/state', 'local/config']) fs.mkdirSync(path.join(root, name), { mode: 0o700 });
@@ -43,7 +44,7 @@ function fixture(t) {
     }
     calls.push([action, id]); return success('accepted', {});
   } };
-  const options = { paths, accessStore: { db }, manager, hub, configuration: { version: 1, enabled: true, idleMs: 1000, pollMs: 100, maxActive: 1 }, clock: () => now };
+  const options = { publishedReader: () => ({ workforce: { employees: () => failure('not_initialized') } }), paths, accessStore: { db }, manager, hub, configuration: { version: 1, enabled: true, idleMs: 1000, pollMs: 100, maxActive: 1 }, clock: () => now };
   function open() { execution = new DirectoryExecution(options); execution.wake = () => {}; }
   open();
   t.after(async () => { await execution.close(); db.close(); fs.rmSync(root, { recursive: true, force: true }); });
@@ -66,6 +67,7 @@ test('idle workers stop, reads never wake them, and concurrent manual requests s
   await c.execution.runPending(); c.advance(1200); await c.execution.runPending();
   assert.equal(c.active.has(id), false);
   assert.equal(c.execution.store.get(id).state, 'sleeping');
+  assert.match(c.execution.store.get(id).operation_id, /^sleep_[a-f0-9]{32}$/);
   const before = c.calls.length;
   for (let i = 0; i < 20; i++) assert.equal((await c.execution.invoke(id, 'sync.status', { id: 'paycom-main-workforce' })).ok, true);
   assert.equal(c.calls.length, before);

core/api/access-http.js

@@ -62,6 +62,7 @@ function createAccessHttp({
     if (!current || request.headers['x-dispatch-csrf'] !== current.csrfToken) throw new AccessError('csrf_invalid', 403);
     requireJson(request);
     requireNoQuery(url);
+    if ((current.dspView || current.user.platformRole !== 'owner') && access.store?.releaseBlocked?.(current.activeOrganizationId)) throw new AccessError('release_busy', 409);
   }
 
   function activeOrganizationId(current) {
@@ -354,11 +355,12 @@ function createAccessHttp({
       } else requireNoQuery(url);
       access.requirePlatform(current, 'platform.installations.manage');
       if (!updates) throw new AccessError('installation_operator_disabled', 503);
+      if (updates.ownerOnly && (current.user.platformRole !== 'owner' || current.dspView)) throw new AccessError('platform_forbidden', 403);
       if (request.method === 'POST') {
         requireMutation(request, current, url);
-        updates.command(current, await readJson(request));
+        await updates.command(current, await readJson(request));
       }
-      sendJson(response, 200, { ok: true, status: 'found', data: updates.view(releaseId), error: null });
+      sendJson(response, 200, { ok: true, status: 'found', data: await updates.view(releaseId), error: null });
       return true;
     }
 

core/api/directory-platform.js

@@ -21,7 +21,6 @@ const { DirectoryLifecycleWorker } = require('../../host/controller/lifecycle');
 const { createDirectoryMonitor } = require('../../host/capacity/monitor');
 const { createDirectoryDiagnostics } = require('../../host/controller/diagnostics');
 const { loadDashboardSettings } = require('../../host/controller/dashboard-settings');
-const { createPlatformUpdates } = require('../accounts/src/platform-updates');
 const { ManualBackups, interruptedRestore } = require('../../host/storage/manual-backups');
 const { invitationDeliveryFromEnvironment } = require('./invitation-email');
 const { DirectoryExecution } = require('../../host/controller/execution');
@@ -34,9 +33,10 @@ async function startDirectoryApi({ paths, installation, host, port = 4310, addre
   if (!['127.0.0.1', '::1', 'localhost'].includes(address) || !Number.isInteger(port) || port < 0 || port > 65535) {
     throw new Error('directory_dashboard_invalid');
   }
-  let store, runtime, worker, execution, server, closing;
+  let store, runtime, worker, execution, server, closing, updateControl;
   const close = () => closing ||= (async () => {
     if (server?.listening) await new Promise(resolve => server.close(resolve));
+    await updateControl?.close();
     await worker?.close();
     await execution?.close();
     try { await runtime?.close(); } finally { store?.close(); }
@@ -98,12 +98,18 @@ async function startDirectoryApi({ paths, installation, host, port = 4310, addre
         for (const result of results) if (result.status === 'rejected') onError(result.reason);
       } });
     const client = createRuntimeAgentDispatchClient({ runtimeKey: 'unassigned', hub: runtime.hub });
-    // Local development has no release feed. History stays readable without
-    // activating legacy downloads, rollout workers or pre-update backups.
-    const updates = createPlatformUpdates({ store, enabled: false });
+    // Configuration opts into independent updates; activation belongs to the
+    // external worker and this controller's scoped DSP lifecycle.
+    updateControl = await require('../updates/directory').directoryUpdates({ paths, store, manager: runtime.manager, execution });
+    const updates = updateControl.service;
     const config = dashboardConfig({});
     const pluginAssets = require('../plugins/assets').createPluginAssets({ dspRoot: id => runtime.manager.checkedDsp(runtime.manager.journal.record(id)).root });
-    server = serverFactory({ access, client, config, operator, paycomSetup, connections, plugins, pluginAssets, publicOrigin, secureCookies, updates, backups,
+    const installedCore = require('../installations/src/release-delivery-files').privateJson(require('../../host/releases/core').receiptFile(paths), process.geteuid(), true);
+    const coreMaintenance = () => {
+      const state = require('../installations/src/release-delivery-files').privateJson(path.join(paths.local, 'state/updates/releases.json'), process.geteuid(), true);
+      return state?.operation?.product === 'core' ? { phase: 'updating', nonce: state.operation.preparation?.nonce } : null;
+    };
+    server = serverFactory({ coreIdentity: installedCore, coreMaintenance, access, client, config, operator, paycomSetup, connections, plugins, pluginAssets, publicOrigin, secureCookies, updates, backups,
       // Public installations require email setup before creating invitations.
       // Loopback-only development can still hand off invitation links manually.
       invitationDelivery,

core/api/http.js

@@ -185,7 +185,7 @@ function publicHttpFailure(error) {
   // These expected availability failures contain no private diagnostic data.
   // Keep every other server failure opaque, including untrusted lookalike errors.
   if (statusCode === 503 && error instanceof AccessError
-      && ['installation_operator_disabled', 'invitation_email_unavailable', 'turnstile_unavailable', 'password_recovery_unavailable', 'password_recovery_busy'].includes(error.code)) {
+      && ['release_worker_unavailable', 'installation_operator_disabled', 'invitation_email_unavailable', 'turnstile_unavailable', 'password_recovery_unavailable', 'password_recovery_busy'].includes(error.code)) {
     return { statusCode, code: error.code };
   }
   if (statusCode >= 500) return { statusCode, code: 'dashboard_unavailable' };

core/auth-broker/server.js

@@ -16,6 +16,9 @@ const { serveBackend } = require('../plugins/transport');
 async function main() {
   process.umask(0o077);
   const paths = loadPlatformPaths(), lock = acquireLock(paths, 'plugin-backend');
+  const definitions = () => require('../plugins/package-catalog').packageCatalog(paths)?.definitions() || [];
+  require('../../shared/plugin-sdk/catalog').configureCatalog(definitions);
+  require('dispatch-protocol/plugin-sdk/catalog').configureCatalog(definitions);
   const databaseRoot = privateDirectory(path.join(paths.local, 'state/access-control'));
   const store = new AccessStore({ databaseRoot, database: path.join(databaseRoot, 'access-control.sqlite3') });
   const journal = new DirectoryJournal(paths), authority = directoryAccessAuthority({ paths, store, journal });

core/plugins/backend.js

@@ -122,7 +122,7 @@ async function openPluginBackend({ paths, installation, store, dspRoot, permitte
         AND p.applied_state='enabled' AND p.revision=p.applied_revision AND i.status='ready'`).all()) {
         try { if (!settingsStore(dspRoot(row.runtime_key),row.plugin_id).pending()) continue;
           const manifest = selected(row.runtime_key,row.plugin_id).manifest.plugin;
-          applySettingsPolicy(dspRoot(row.runtime_key),manifest); } catch { /* Pending policy is retried after lifecycle/storage recovery. */ }
+          if (!require('../../host/releases/guard').updating(paths, row.runtime_key)) applySettingsPolicy(dspRoot(row.runtime_key),manifest); } catch { /* Pending policy is retried after lifecycle/storage recovery. */ }
       }
       }
     } finally { reaping = false; }