diff --git a/.changeset/21489-job-bodies-install-local.md b/.changeset/21489-job-bodies-install-local.md new file mode 100644 index 0000000000..cf178b5a77 --- /dev/null +++ b/.changeset/21489-job-bodies-install-local.md @@ -0,0 +1,23 @@ +--- +'@objectstack/runtime': minor +'@objectstack/cloud-connection': minor +'@objectstack/cli': minor +'@objectstack/spec': minor +--- + +fix(runtime,cloud-connection)!: a job's sandboxed `body` is scheduled on every door that brings an artifact in, and install-local refuses an enabled job with no `body` (#21489) + +Clause-②: yes (narrowing) + + + +**BREAKING**: `os package install` (the install-local door, `POST /api/v1/marketplace/install-local`) now refuses a package that declares an **enabled job with no `body`**. Such a job names its code only through `handler` — a `defineStack({ functions })` entry, which travels in the artifact's runtime module and never in the package JSON this door installs — so it used to install with a 200 and never run, hot or after a restart, with nothing saying so. + +- **Job bodies run.** A job's sandboxed `body` (`JobSchema.body`, the hook body shape) is now scheduled on every door that brings an artifact in: the boot (`os start --artifact`, a `defineStack` config) and install-local, on install and on every rehydrate after a restart. One binder does it for all of them. With both `body` and `handler` declared, the `body` wins. The body runs in the QuickJS sandbox with `ctx.api` (as system: a job has no caller), `ctx.log` and `ctx.crypto` behind its declared `capabilities`. The job's `timeoutMs` is its one time limit; with none, a job body gets a 5000 ms CPU budget. A body may return `{ outcome: 'degraded', reason }` to report a run that did not do its work. +- **A package's jobs stop with it.** Re-scheduling a package's jobs replaces its set: a reinstall whose new version drops, disables or can no longer run a job cancels that job, and a version with no jobs cancels them all. Uninstalling a package cancels its scheduled jobs through a new uninstall cleanup, `runtime.package-jobs`, on the protocol's uninstall-cleanup registry, so install-local's `DELETE` and the protocol's package uninstall both stop them and report it in `cleanups`. Another package's jobs are never touched. +- **The refusal.** The install answers `422` with `VALIDATION_ERROR`, names each refused job and the function its `handler` declares, and installs nothing: nothing is registered, persisted or scheduled. A disabled job (`enabled: false`) is not judged. A package installed by an earlier version keeps rehydrating; its handler-only job is reported at `warn` and does not run. +- **CLI.** `os package install` prints a refusal's code beside its status (`Install failed (422 VALIDATION_ERROR): …`), for every refusal alike. +- **Spec.** The shipped liveness ledger records `job.body` (`language`, `source`, `capabilities`, `memoryMb`) as live, so `os validate` / `os build` no longer warn that a job's `body` is planned and not read yet. `body.timeoutMs` stays refused on a job. `JobSchema.body`'s description and the `defineJob` example no longer say to keep a `handler` until the runtime runs job bodies. +- **Unchanged:** a `handler` job on a boot that loads the artifact's runtime module (`os start --artifact`, a `defineStack` config) still runs its `functions` entry; a package without jobs installs exactly as before. + +The route for a refused package: give each enabled job a `body` (sandboxed JS that reaches data through `ctx.api`), or boot the artifact with `os start --artifact`, which loads its runtime module. It ships as `minor` under the launch-window convention for accept-set narrowings. diff --git a/content/docs/automation/jobs.mdx b/content/docs/automation/jobs.mdx index 1cb22ad204..245a2b1977 100644 --- a/content/docs/automation/jobs.mdx +++ b/content/docs/automation/jobs.mdx @@ -159,18 +159,21 @@ export const CloseStaleTasksJob = defineJob({ `, capabilities: ['api.read', 'api.write', 'log'], }, - handler: 'closeStaleTasks', // deprecated — kept until the runtime runs job bodies (see below) + handler: 'closeStaleTasks', // deprecated and optional — `body` wins when both are present timeoutMs: 120000, }); ``` - - **The runtime does not run a job `body` yet.** Scheduling a job's body is a - separate change that has not landed: until it does, a job is scheduled through - its `handler`, and a job with a `body` and no `handler` is skipped at boot with - a `warn`. `os validate` and `os build` say so wherever a `body` is set. Keep - `handler` beside `body` for now; the `body` is validated today, so it is ready - the day the runtime starts honouring it. + + **Every door runs a job `body`.** The boot (a config, or `os start --artifact`) + and `os package install` (on install, and again on every restart) schedule a + job's `body` through the same binder. A `handler` is code: it travels only in the + artifact's runtime module, so it runs only on a boot that loads that module (a + config, or `os start --artifact`). `os package install` therefore refuses a + package whose enabled job has no `body`, with `422 VALIDATION_ERROR` and the + remedy: give the job a `body`, or boot it with `os start --artifact`. + Uninstalling a package stops its scheduled jobs at once, and a reinstall whose + new version drops a job stops that job. What running in the sandbox means for the code in `source`: diff --git a/content/docs/references/system/job.mdx b/content/docs/references/system/job.mdx index 7e13bd5fb3..cf802d57bf 100644 --- a/content/docs/references/system/job.mdx +++ b/content/docs/references/system/job.mdx @@ -58,7 +58,7 @@ const result = CronScheduleSchema.parse(data); | **description** | `string` | optional | Job description / purpose | | **schedule** | `{ type: 'cron'; expression: string \| object; timezone?: string } \| { type: 'interval'; intervalMs: integer } \| { type: 'once'; at: string }` | ✅ | Job schedule configuration | | **handler** | `string` | optional | Handler function name (must match a key in `defineStack({ functions })`) — DEPRECATED, prefer `body`. When both are present `body` wins; a job must declare one of the two. | -| **body** | `{ language: 'js'; source: string; capabilities?: Enum<'api.read' \| 'api.write' \| 'api.transaction' \| 'crypto.uuid' \| 'log'>[]; timeoutMs?: integer; … }` | optional | Job body — a sandboxed JS (L2) body, the same shape hooks and actions use; an expression (L1) body is refused, because a job runs for its effects and an expression has none. Preferred over `handler`: when both are present `body` wins. It runs in the QuickJS sandbox with no module scope (no imports, no helpers or constants from the surrounding file): it reaches data only through `ctx.api` under its declared `capabilities` (`api.read` / `api.write` / `api.transaction`) and logs through `ctx.log` (`log`); the in-process handler context (`ql`, `logger`, `bundle`) does not exist there. Its time limit is the job's `timeoutMs` (see there): long-running work declares a `timeoutMs` that covers it, or splits into bounded runs that each finish within it. The runtime binder that schedules job bodies has not landed yet: until it does a job runs through `handler`, so keep `handler` beside `body`. | +| **body** | `{ language: 'js'; source: string; capabilities?: Enum<'api.read' \| 'api.write' \| 'api.transaction' \| 'crypto.uuid' \| 'log'>[]; timeoutMs?: integer; … }` | optional | Job body — a sandboxed JS (L2) body, the same shape hooks and actions use; an expression (L1) body is refused, because a job runs for its effects and an expression has none. Preferred over `handler`: when both are present `body` wins. It runs in the QuickJS sandbox with no module scope (no imports, no helpers or constants from the surrounding file): it reaches data only through `ctx.api` under its declared `capabilities` (`api.read` / `api.write` / `api.transaction`) and logs through `ctx.log` (`log`); the in-process handler context (`ql`, `logger`, `bundle`) does not exist there. Its time limit is the job's `timeoutMs` (see there): long-running work declares a `timeoutMs` that covers it, or splits into bounded runs that each finish within it. Every door that brings an artifact in schedules a job's `body` — the boot, and `os package install` on install and on every restart — while a `handler` is code that travels only in the artifact's runtime module and runs only on a boot that loads it (a config, or `os start --artifact`); `os package install` therefore refuses an enabled job with no `body`. | | **retryPolicy** | `{ maxRetries?: integer; backoffMs?: integer; backoffMultiplier?: number; maxRetryDelayMs?: integer; … }` | optional | Retry policy: failed runs (including timeouts) are retried with exponential backoff (delay = min(backoffMs * backoffMultiplier^(retry-1), maxRetryDelayMs), optionally jittered) up to maxRetries retries after the initial attempt. Omit the block for a single attempt; declaring it without `maxRetries` also means no retry since 17.0.0 — state a count to opt in. | | **timeoutMs** | `integer` | optional | Per-attempt time limit in milliseconds; an over-limit run is recorded with execution status "timeout". A `handler` run is abandoned, not forcibly cancelled. For a job with a `body` this is the ONE time limit: one attempt is one sandbox invocation, the runtime bounds that invocation by this value, and the body shape's own `timeoutMs` (capped at 30000 for hooks and actions) is refused on a job — so this key, which has no such cap, is where long-running work states how long it needs. Omit for no per-attempt limit; a `body` run is then still bounded by the sandbox's own default invocation limits. | | **timeout** | `never` | optional | [REMOVED] `job.timeout` was removed in @objectstack/spec 17 — its unit (milliseconds) lived only in the description while the sibling `retryPolicy.backoffMs` spells its own, so the same number read as two conventions on one surface. Rename the key to `timeoutMs`; the value (milliseconds) is unchanged. Run `os migrate meta --from 17` to list the mechanical edits for existing sources; apply them by hand. | diff --git a/docs/qa/platform-checklist/areas/integration-system.json b/docs/qa/platform-checklist/areas/integration-system.json index 537b4a026c..301935371b 100644 --- a/docs/qa/platform-checklist/areas/integration-system.json +++ b/docs/qa/platform-checklist/areas/integration-system.json @@ -730,7 +730,7 @@ "source": [ "packages/spec/src/system/job.zod.ts#ScheduleSchema (ScheduleSchema discriminated union; JOB_ID_RETIRED; retryPolicy/timeoutMs docs incl. the 17.0.0 maxRetries default flip #4661; JobExecutionStatus)", "packages/spec/liveness/job.json (per-prop verdicts + the #4509 closed-door rationale)", - "packages/runtime/src/app-plugin.ts#handler (registration, enabled/handler skip lines)", + "packages/runtime/src/app-artifact-handlers.ts#scheduleAppArtifactJobs (registration, enabled/handler skip lines — the binder's job half, which every door that brings an artifact in calls)", "packages/services/service-job/src/cron-job-adapter.ts + db-job-adapter.ts (all three schedule shapes; sys_job/sys_job_run persistence) + run-with-policy.ts (retry/timeout enforcement, #3494)", "examples/app-showcase/src/automation/jobs/index.ts#showcase_health_sweep (showcase_health_sweep fixture + its #4774/#4888 history)" ], diff --git a/packages/cli/src/commands/package/install.ts b/packages/cli/src/commands/package/install.ts index cce12e57d2..1cbe3806d4 100644 --- a/packages/cli/src/commands/package/install.ts +++ b/packages/cli/src/commands/package/install.ts @@ -205,7 +205,15 @@ export default class PackageInstall extends Command { 'MarketplaceInstallLocalPlugin (see @objectstack/cloud-connection).', ); } else { - printError(`Install failed (${res.status}): ${res.error}`); + // [#21489] The runtime's refusal is printed with its CODE beside the + // status, for every refusal alike — the code is the machine-readable + // half an installer (human or AI) branches on, and the message carries + // the remedy (e.g. a package whose enabled job has no `body`: + // `422 VALIDATION_ERROR`, "give the job a `body`, or boot it with + // `os start --artifact`"). No case per code: a refusal this command + // has never heard of renders the same way. + const code = typeof res.body?.error?.code === 'string' ? ` ${res.body.error.code}` : ''; + printError(`Install failed (${res.status}${code}): ${res.error}`); } this.exit(1); return; diff --git a/packages/cli/test/package-install-local-jobs.integration.test.ts b/packages/cli/test/package-install-local-jobs.integration.test.ts new file mode 100644 index 0000000000..f3f571aa3e --- /dev/null +++ b/packages/cli/test/package-install-local-jobs.integration.test.ts @@ -0,0 +1,555 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #21489 — a package's job BODIES run on every door that brings an artifact + * in, and install-local refuses the one job shape no JSON door can run. + * + * ## The defect, measured at this door before the fix + * + * A job's runnable code was only ever a `handler`: the name of a + * `defineStack({ functions })` entry, whose callable travels in the artifact's + * runtime module, which only `os start --artifact` imports. So a package + * installed with `os package install` into a running platform: + * + * - declaring a job with a sandboxed `body` (`JobSchema.body`, the hook body + * shape) was never scheduled — not hot, not after a restart, and not even + * on an `--artifact` boot, because the boot resolved `handler` alone; + * - declaring a job with only a `handler` installed "successfully" and was + * never scheduled either — the install answered 200 and nothing said the + * job would never run. + * + * ## What each `it` reads + * + * One host runtime (`requires: ['job']`, package scheduled work switched on), + * three boots: + * + * 1. INSTALL — the body-job package installs and its body runs on its + * schedule (the rows it writes appear); the handler-only package is + * REFUSED by the install door with its code and remedy, and nothing of it + * is installed; + * 2. RESTART — same home: the ledger rehydrate schedules the body job again + * (new rows appear after the restart); + * 3. CONTROL — `os start --artifact` of one artifact carrying a body job and + * a handler job (with its runtime module): both run. + * + * Every reading goes through a door a user uses: the CLI's own output and exit + * code for the install, the data route for the rows a job wrote. + * + * ## Spawn shape + * + * Shared with `package-install-local-handlers.integration.test.ts` (#21321) + * and `package-install-local-boot-steps.integration.test.ts` (#21322): the tsx + * source entry, one process group per `os start`, every workspace package — + * `@objectstack/runtime` and `@objectstack/cloud-connection` included — + * resolved through its `exports` to `dist/`, so an ablation of either + * package's source reaches this file only after that package is rebuilt. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { spawn, type ChildProcess } from 'node:child_process'; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + CLI, + childEnv, + E2E_SECRET_KEY, + portContentionError, + portDriftError, + probeThroughChild, + randomPort, + TSX, +} from './helpers/serve-process.js'; + +/** The banner's tail — every row above it has printed. */ +const READY = /Press Ctrl\+C to stop/; +const BOOT_TIMEOUT_MS = 180_000; +/** How long a phase waits for a 1-second interval job to have written a row. */ +const RUN_WAIT_MS = 20_000; + +/** The development dev-admin seed — see the #21321 sibling for why it is the operator on every boot. */ +const EMAIL = 'admin@objectos.ai'; +const PASSWORD = 'admin123'; + +/** + * The HOST's own object, where every installed package's job writes its rows. + * Not the package's own object: an uninstall withdraws the package from the + * running kernel (#21576), so its object stops answering — a job still running + * after the uninstall would fail its write there and leave no row to see. The + * host's object outlives every package, so a run that should have stopped shows. + */ +const HOST_TICK = 'host_tick'; + +const BODY_APP_ID = 'com.example.jobsapp'; +const BODY_TICK = 'jobs_app_tick'; +const BODY_JOB = 'jobs_app_tick_body'; + +/** Another installed package — the control a package's uninstall or reinstall must leave running. */ +const OTHER_APP_ID = 'com.example.otherjobs'; +const OTHER_TICK = 'other_jobs_tick'; +const OTHER_JOB = 'other_jobs_tick_body'; + +/** A package reinstalled with a version that DROPS one of its two jobs. */ +const DROP_APP_ID = 'com.example.dropjobs'; +const DROP_TICK = 'drop_jobs_tick'; +const DROP_KEPT = 'drop_jobs_kept'; +const DROP_GONE = 'drop_jobs_gone'; + +/** "Writes no further row": let an in-flight run land, take the floor, then read again this much later. */ +const SETTLE_MS = 1_500; +const QUIET_WAIT_MS = 4_000; + +const HANDLER_APP_ID = 'com.example.handlerjobs'; +const HANDLER_TICK = 'handler_jobs_tick'; +const HANDLER_JOB = 'handler_jobs_tick_handler'; + +/** A job with a sandboxed body that writes one row per run into `object`. */ +function bodyJob(name: string, object: string) { + return { + name, + schedule: { type: 'interval', intervalMs: 1000 }, + body: { + language: 'js', + capabilities: ['api.write'], + source: `await ctx.api.object('${object}').insert({ name: '${name}' });`, + }, + timeoutMs: 10_000, + enabled: true, + }; +} + +/** A job whose code is a function NAME only — the deprecated form. */ +function handlerJob(name: string) { + return { name, schedule: { type: 'interval', intervalMs: 1000 }, handler: 'tick', enabled: true }; +} + +function tickObject(name: string) { + return { + name, + label: 'Tick', + sharingModel: 'public_read_write', + fields: { name: { type: 'text', label: 'Name' } }, + }; +} + +/** The body-job package, as `os build` writes `dist/objectstack.json` (schema defaults trimmed). */ +const BODY_ARTIFACT = { + manifest: { id: BODY_APP_ID, namespace: 'jobs_app', version: '0.1.0', type: 'app', name: 'Jobs App' }, + objects: [tickObject(BODY_TICK)], + jobs: [bodyJob(BODY_JOB, HOST_TICK)], +}; + +const OTHER_ARTIFACT = { + manifest: { id: OTHER_APP_ID, namespace: 'other_jobs', version: '0.1.0', type: 'app', name: 'Other Jobs' }, + objects: [tickObject(OTHER_TICK)], + jobs: [bodyJob(OTHER_JOB, HOST_TICK)], +}; + +/** One version of the drop package, declaring `jobs` (all body jobs into one object). */ +function dropArtifact(version: string, jobs: string[]) { + return { + manifest: { id: DROP_APP_ID, namespace: 'drop_jobs', version, type: 'app', name: 'Drop Jobs' }, + objects: [tickObject(DROP_TICK)], + jobs: jobs.map((name) => bodyJob(name, HOST_TICK)), + }; +} + +/** The handler-only package: its one enabled job names a function no JSON door carries. */ +const HANDLER_ARTIFACT = { + manifest: { id: HANDLER_APP_ID, namespace: 'handler_jobs', version: '0.1.0', type: 'app', name: 'Handler Jobs' }, + objects: [tickObject(HANDLER_TICK)], + jobs: [handlerJob(HANDLER_JOB)], +}; + +/** + * The CONTROL artifact: both job forms in one boot artifact, the handler's + * callable in the runtime module `os start --artifact` merges + * (`mergeRuntimeModule`, the path `os build` emits). + */ +const CONTROL_ARTIFACT = { + manifest: { id: 'com.example.controljobs', namespace: 'control_jobs', version: '0.1.0', type: 'app', name: 'Control Jobs' }, + objects: [tickObject(BODY_TICK), tickObject(HANDLER_TICK)], + jobs: [bodyJob(BODY_JOB, BODY_TICK), handlerJob(HANDLER_JOB)], + runtimeModule: './runtime.mjs', +}; +const CONTROL_RUNTIME_MODULE = + 'export const functions = {\n' + + ' tick: async ({ ql }) => {\n' + + ` await ql.insert('${HANDLER_TICK}', { name: '${HANDLER_JOB}' }, { context: { isSystem: true } });\n` + + ' },\n' + + '};\n'; + +/** + * The RUNTIME the packages are installed into. `os start` composes its + * services from the boot stack's `requires`, never from a package installed + * later, so the host declares the job service and nothing else. + */ +const HOST_ARTIFACT = { + manifest: { id: 'com.example.host', namespace: 'host', version: '0.1.0', type: 'app', name: 'Host' }, + requires: ['job'], + objects: [tickObject(HOST_TICK)], +}; + +const groups: ChildProcess[] = []; +const dirs: string[] = []; + +interface LiveStart { + child: ChildProcess; + base: string; + output: () => string; +} + +function bootStart(cwd: string, home: string, port: string, extra: string[] = []): Promise { + return new Promise((resolveBoot, rejectBoot) => { + const child = spawn(TSX, [CLI, 'start', '-p', port, '--home', home, '--auth-secret', E2E_SECRET_KEY, '--no-ui', ...extra], { + cwd, + // `childEnv`, never a bare `...process.env` — see its header. Package + // scheduled work is OFF by default in every posture (#17396); this + // runtime is one that runs it. + env: childEnv({ + NO_COLOR: '1', + OS_CLOUD_URL: 'off', + OS_LOG_LEVEL: 'warn', + OS_SECRET_KEY: E2E_SECRET_KEY, + OS_AUTOMATION_SCHEDULED_WORK_ENABLED: 'true', + }), + stdio: ['ignore', 'pipe', 'pipe'], + // Own process group: `os start` supervises a `serve` grandchild. + detached: true, + }); + groups.push(child); + let out = ''; + let settled = false; + const settle = (err: Error | null) => { + if (settled) return; + settled = true; + clearTimeout(timer); + if (err) rejectBoot(err); + else resolveBoot({ child, base: `http://localhost:${port}`, output: () => out }); + }; + const timer = setTimeout( + () => settle(new Error(`os start never printed ${READY}\n--- output ---\n${out.slice(-4000)}`)), + BOOT_TIMEOUT_MS, + ); + const onData = (d: unknown) => { + out += String(d); + // The child is the authority on the port it bound. + if (READY.test(out)) settle(portDriftError(out, 'os start', port)); + }; + child.stdout?.on('data', onData); + child.stderr?.on('data', onData); + child.on('exit', (code) => + settle(portContentionError(out, 'os start', port) + ?? new Error(`os start exited ${String(code)} before ${READY}\n--- output ---\n${out.slice(-4000)}`)), + ); + }); +} + +async function stopGroup(child: ChildProcess): Promise { + if (child.pid === undefined || child.exitCode !== null || child.signalCode !== null) return; + await new Promise((done) => { + const give = setTimeout(() => { + try { process.kill(-child.pid!, 'SIGKILL'); } catch { /* group already gone */ } + done(); + }, 15_000); + child.once('exit', () => { clearTimeout(give); done(); }); + try { process.kill(-child.pid!, 'SIGTERM'); } catch { clearTimeout(give); done(); } + }); +} + +interface Answer { status: number; body: any } + +/** One exchange against the running `os start`, attributed to the child if the transport fails. ⛔ No assertion inside it. */ +function http(live: LiveStart, method: string, path: string, token: string, body?: unknown): Promise { + return probeThroughChild( + { + child: live.child, + transcript: () => `\n--- child output ---\n${live.output().slice(-4000)}`, + label: 'package-install-local-jobs', + what: `${method} ${path}`, + }, + async () => { + const r = await fetch(`${live.base}${path}`, { + method, + headers: { + origin: live.base, + ...(body !== undefined ? { 'content-type': 'application/json' } : {}), + ...(token ? { authorization: `Bearer ${token}` } : {}), + }, + ...(body !== undefined ? { body: JSON.stringify(body) } : {}), + }); + const text = await r.text(); + let parsed: any = text; + try { parsed = JSON.parse(text); } catch { /* keep the text */ } + return { status: r.status, body: parsed }; + }, + ); +} + +async function authenticate(live: LiveStart): Promise { + const res = await http(live, 'POST', '/api/v1/auth/sign-in/email', '', { email: EMAIL, password: PASSWORD }); + const token = res.body?.token; + if (res.status !== 200 || typeof token !== 'string') { + throw new Error(`auth answered ${res.status}: ${JSON.stringify(res.body)}\n--- output ---\n${live.output().slice(-3000)}`); + } + return token; +} + +/** + * `os package install ./dist/objectstack.json` against the running runtime. + * ⛔ Asynchronous on purpose — see the #21321 sibling: a `spawnSync` stops this + * process draining the server's pipes for the whole install. + */ +function packageInstall(appDir: string, live: LiveStart): Promise<{ exit: number | null; output: string }> { + return new Promise((done) => { + const child = spawn(TSX, [CLI, 'package', 'install', './dist/objectstack.json', '--runtime', live.base, '--email', EMAIL, '--password', PASSWORD], { + cwd: appDir, + env: childEnv({ NO_COLOR: '1' }), + stdio: ['ignore', 'pipe', 'pipe'], + }); + let output = ''; + child.stdout?.on('data', (d) => { output += String(d); }); + child.stderr?.on('data', (d) => { output += String(d); }); + const timer = setTimeout(() => child.kill('SIGKILL'), 120_000); + child.on('close', (code) => { clearTimeout(timer); done({ exit: code, output }); }); + }); +} + +/** The rows of a `GET /api/v1/data/:object` list answer, whichever envelope it came in. */ +function rowsOf(answer: Answer): any[] { + const b = answer.body?.data ?? answer.body; + if (Array.isArray(b)) return b; + if (Array.isArray(b?.records)) return b.records; + if (Array.isArray(b?.items)) return b.items; + return []; +} + +/** The rows `job` has written into `object`, read through the data route. */ +async function jobRows(live: LiveStart, token: string, object: string, job: string): Promise { + return http(live, 'GET', `/api/v1/data/${object}?name=${encodeURIComponent(job)}&limit=500`, token); +} + +/** + * Wait (bounded) until `job` has written MORE than `floor` rows into + * `object` — a 1-second interval job is read for a while rather than once, so + * a run a beat after the boot is not misread as one that never happens. + */ +async function awaitRuns(live: LiveStart, token: string, object: string, job: string, floor: number): Promise<{ answer: Answer; floor: number }> { + const deadline = Date.now() + RUN_WAIT_MS; + let answer = await jobRows(live, token, object, job); + while (rowsOf(answer).length <= floor && Date.now() < deadline) { + await new Promise((r) => setTimeout(r, 500)); + answer = await jobRows(live, token, object, job); + } + return { answer, floor }; +} + +const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); + +/** + * The reading for an act that must STOP `job`: let a run already in flight + * land, take the row count as the floor, wait, and count again. A job that + * was stopped leaves `after === floor`; one still scheduled (every second) + * adds rows in between. + */ +async function quietAfter(live: LiveStart, token: string, object: string, job: string): Promise<{ floor: number; after: number }> { + await sleep(SETTLE_MS); + const floor = rowsOf(await jobRows(live, token, object, job)).length; + await sleep(QUIET_WAIT_MS); + const after = rowsOf(await jobRows(live, token, object, job)).length; + return { floor, after }; +} + +interface InstallRun { exit: number | null; output: string } +const readings: { + bodyInstall?: InstallRun; + handlerInstall?: InstallRun; + installed?: Answer; + afterInstall?: { answer: Answer; floor: number }; + afterRestart?: { answer: Answer; floor: number }; + otherInstall?: InstallRun; + dropInstall?: InstallRun; + dropReinstall?: InstallRun; + dropBefore?: { answer: Answer; floor: number }; + dropGoneHot?: { floor: number; after: number }; + dropKeptHot?: { answer: Answer; floor: number }; + uninstall?: Answer; + bodyGoneHot?: { floor: number; after: number }; + otherHot?: { answer: Answer; floor: number }; + bodyGoneRestart?: { floor: number; after: number }; + dropGoneRestart?: { floor: number; after: number }; + dropKeptRestart?: { answer: Answer; floor: number }; + controlBody?: { answer: Answer; floor: number }; + controlHandler?: { answer: Answer; floor: number }; + output: Record; +} = { output: {} }; + +beforeAll(async () => { + const root = mkdtempSync(join(tmpdir(), 'install-local-jobs-')); + dirs.push(root); + const write = (dir: string, artifact: unknown) => { + mkdirSync(join(dir, 'dist'), { recursive: true }); + writeFileSync(join(dir, 'dist', 'objectstack.json'), JSON.stringify(artifact, null, 2), 'utf8'); + }; + const bodyApp = join(root, 'body-app'); + const handlerApp = join(root, 'handler-app'); + const controlApp = join(root, 'control-app'); + write(bodyApp, BODY_ARTIFACT); + write(handlerApp, HANDLER_ARTIFACT); + write(controlApp, CONTROL_ARTIFACT); + const otherApp = join(root, 'other-app'); + const dropAppV1 = join(root, 'drop-app-v1'); + const dropAppV2 = join(root, 'drop-app-v2'); + write(otherApp, OTHER_ARTIFACT); + write(dropAppV1, dropArtifact('0.1.0', [DROP_KEPT, DROP_GONE])); + write(dropAppV2, dropArtifact('0.2.0', [DROP_KEPT])); + writeFileSync(join(controlApp, 'dist', 'runtime.mjs'), CONTROL_RUNTIME_MODULE, 'utf8'); + + // The runtime boots the HOST artifact — never a package — so the packages + // reach it only through the install. + const runtimeDir = join(root, 'runtime'); + mkdirSync(runtimeDir, { recursive: true }); + const hostArtifact = join(runtimeDir, 'host.json'); + writeFileSync(hostArtifact, JSON.stringify(HOST_ARTIFACT, null, 2), 'utf8'); + const home = join(runtimeDir, 'home'); + const port = randomPort(); + + // ── boot 1: the host, hot installs, the body job runs ───────────────── + const first = await bootStart(runtimeDir, home, port, ['--artifact', hostArtifact]); + const token = await authenticate(first); + readings.bodyInstall = await packageInstall(bodyApp, first); + readings.otherInstall = await packageInstall(otherApp, first); + readings.dropInstall = await packageInstall(dropAppV1, first); + readings.handlerInstall = await packageInstall(handlerApp, first); + readings.installed = await http(first, 'GET', '/api/v1/marketplace/install-local', token); + readings.afterInstall = await awaitRuns(first, token, HOST_TICK, BODY_JOB, 0); + + // Reinstall the drop package with a version that no longer declares DROP_GONE. + readings.dropBefore = await awaitRuns(first, token, HOST_TICK, DROP_GONE, 0); + readings.dropReinstall = await packageInstall(dropAppV2, first); + readings.dropGoneHot = await quietAfter(first, token, HOST_TICK, DROP_GONE); + readings.dropKeptHot = await awaitRuns(first, token, HOST_TICK, DROP_KEPT, + rowsOf(await jobRows(first, token, HOST_TICK, DROP_KEPT)).length); + + // Uninstall the body package; the other package is the control. + readings.uninstall = await http(first, 'DELETE', `/api/v1/marketplace/install-local/${BODY_APP_ID}`, token); + readings.bodyGoneHot = await quietAfter(first, token, HOST_TICK, BODY_JOB); + readings.otherHot = await awaitRuns(first, token, HOST_TICK, OTHER_JOB, + rowsOf(await jobRows(first, token, HOST_TICK, OTHER_JOB)).length); + readings.output.install = first.output(); + await stopGroup(first.child); + + // ── boot 2: same host, home and cwd — the ledger rehydrates on kernel:ready ── + const second = await bootStart(runtimeDir, home, port, ['--artifact', hostArtifact]); + const token2 = await authenticate(second); + // The rows boot 1 left behind are the floors: only a run in THIS boot lifts one. + const floorOf = async (object: string, job: string) => rowsOf(await jobRows(second, token2, object, job)).length; + const bodyFloor = await floorOf(HOST_TICK, BODY_JOB); + const goneFloor = await floorOf(HOST_TICK, DROP_GONE); + const restartedAt = Date.now(); + readings.afterRestart = await awaitRuns(second, token2, HOST_TICK, OTHER_JOB, await floorOf(HOST_TICK, OTHER_JOB)); + readings.dropKeptRestart = await awaitRuns(second, token2, HOST_TICK, DROP_KEPT, await floorOf(HOST_TICK, DROP_KEPT)); + await sleep(Math.max(0, QUIET_WAIT_MS - (Date.now() - restartedAt))); + readings.bodyGoneRestart = { floor: bodyFloor, after: await floorOf(HOST_TICK, BODY_JOB) }; + readings.dropGoneRestart = { floor: goneFloor, after: await floorOf(HOST_TICK, DROP_GONE) }; + readings.output.restart = second.output(); + await stopGroup(second.child); + + // ── boot 3: the CONTROL — both job forms in the boot artifact ───────── + const controlHome = join(controlApp, 'home'); + const third = await bootStart(controlApp, controlHome, port, ['--artifact', join(controlApp, 'dist', 'objectstack.json')]); + const token3 = await authenticate(third); + readings.controlHandler = await awaitRuns(third, token3, HANDLER_TICK, HANDLER_JOB, 0); + readings.controlBody = await awaitRuns(third, token3, BODY_TICK, BODY_JOB, 0); + readings.output.control = third.output(); + await stopGroup(third.child); +}, 3 * BOOT_TIMEOUT_MS + 12 * RUN_WAIT_MS); + +afterAll(async () => { + for (const child of groups) await stopGroup(child); + for (const dir of dirs) rmSync(dir, { recursive: true, force: true }); +}, 60_000); + +const transcript = (phase: string) => `\n--- ${phase} output ---\n${(readings.output[phase] ?? '').slice(-3000)}`; + +describe('#21489: install-local runs job bodies and refuses handler-only jobs', () => { + it('the body-job package installs (exit 0)', () => { + const run = readings.bodyInstall!; + expect(run.exit, run.output).toBe(0); + expect(run.output).toMatch(/Package installed into the running kernel/); + }); + + it('after install, the body job runs on its schedule — hot, with no restart', () => { + const { answer } = readings.afterInstall!; + expect(answer.status, JSON.stringify(answer.body)).toBe(200); + expect(rowsOf(answer).length, `the installed body job never ran${transcript('install')}`).toBeGreaterThan(0); + }); + + it("after restart, a rehydrated package's body job runs again", () => { + expect(readings.otherInstall!.exit, readings.otherInstall!.output).toBe(0); + const { answer, floor } = readings.afterRestart!; + expect(answer.status, JSON.stringify(answer.body)).toBe(200); + expect(rowsOf(answer).length, `no run after the restart${transcript('restart')}`).toBeGreaterThan(floor); + }); + + it("uninstall: the DELETE answers 200, and the uninstalled package's body job writes no further row — hot", () => { + expect(readings.uninstall!.status, JSON.stringify(readings.uninstall!.body)).toBe(200); + const { floor, after } = readings.bodyGoneHot!; + expect(floor, 'precondition: the job had run before the uninstall').toBeGreaterThan(0); + expect(after, `the uninstalled package's job kept running${transcript('install')}`).toBe(floor); + }); + + it('… and none after a restart', () => { + const { floor, after } = readings.bodyGoneRestart!; + expect(after, `the uninstalled package's job ran after the restart${transcript('restart')}`).toBe(floor); + }); + + it("control: another package's job keeps running across that uninstall", () => { + const { answer, floor } = readings.otherHot!; + expect(rowsOf(answer).length, `the control package's job stopped${transcript('install')}`).toBeGreaterThan(floor); + }); + + it('reinstall: a job the new version DROPPED writes no further row — hot, and none after a restart', () => { + expect(readings.dropInstall!.exit, readings.dropInstall!.output).toBe(0); + expect(readings.dropReinstall!.exit, readings.dropReinstall!.output).toBe(0); + expect(rowsOf(readings.dropBefore!.answer).length, 'precondition: the dropped job had run').toBeGreaterThan(0); + const hot = readings.dropGoneHot!; + expect(hot.after, `the dropped job kept running after the reinstall${transcript('install')}`).toBe(hot.floor); + const restart = readings.dropGoneRestart!; + expect(restart.after, `the dropped job ran after the restart${transcript('restart')}`).toBe(restart.floor); + }); + + it('… while the job the new version KEPT keeps running, hot and after a restart', () => { + const hot = readings.dropKeptHot!; + expect(rowsOf(hot.answer).length, `the kept job stopped${transcript('install')}`).toBeGreaterThan(hot.floor); + const restart = readings.dropKeptRestart!; + expect(rowsOf(restart.answer).length, `the kept job did not run after the restart${transcript('restart')}`).toBeGreaterThan(restart.floor); + }); + + it('the handler-only package is REFUSED, with its code and remedy, and nothing of it is installed', () => { + const run = readings.handlerInstall!; + expect(run.exit, run.output).toBe(1); + // The CLI names the code the runtime answered with, and the remedy. + expect(run.output).toMatch(/Install failed \(422 VALIDATION_ERROR\)/); + expect(run.output).toContain(HANDLER_JOB); + expect(run.output).toMatch(/give the job a `body`/i); + expect(run.output).toMatch(/os start --artifact/); + const listing = readings.installed!; + expect(listing.status, JSON.stringify(listing.body)).toBe(200); + const ids = JSON.stringify(listing.body); + expect(ids).toContain(BODY_APP_ID); + expect(ids, 'a refused package must leave nothing in the ledger').not.toContain(HANDLER_APP_ID); + }); + + it('control: an `--artifact` boot runs the handler job (its runtime module) …', () => { + const { answer } = readings.controlHandler!; + expect(answer.status, JSON.stringify(answer.body)).toBe(200); + expect(rowsOf(answer).length, `the control handler job never ran${transcript('control')}`).toBeGreaterThan(0); + }); + + it('… and the body job, through the same binder', () => { + const { answer } = readings.controlBody!; + expect(answer.status, JSON.stringify(answer.body)).toBe(200); + expect(rowsOf(answer).length, `the control body job never ran${transcript('control')}`).toBeGreaterThan(0); + }); +}); diff --git a/packages/cli/test/package-install-refusal-rendering.test.ts b/packages/cli/test/package-install-refusal-rendering.test.ts new file mode 100644 index 0000000000..add556ffcc --- /dev/null +++ b/packages/cli/test/package-install-refusal-rendering.test.ts @@ -0,0 +1,80 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #21489 — `os package install` prints a runtime refusal with its CODE beside + * the status, and the message (which carries the remedy) verbatim. + * + * The refusal that motivated it: install-local refuses a package whose enabled + * job has no `body` with `422 VALIDATION_ERROR` and a message naming the job and + * both remedies. Before this, the generic branch printed `Install failed (422): + * ` — the machine-readable half an installer (human or AI) branches on + * was dropped. The rendering is GENERIC on purpose: no case per code, so a + * refusal this command has never heard of renders the same way, and an envelope + * with no code renders the status alone, never an invented one. + */ + +import { describe, it, expect, afterEach, vi } from 'vitest'; +import PackageInstall from '../src/commands/package/install.js'; + +/** Stub the runtime's install POST with a refusal envelope. */ +function stubRefusal(status: number, error: unknown): void { + vi.stubGlobal('fetch', vi.fn(async () => ({ + ok: false, + status, + statusText: 'Refused', + headers: { get: () => null }, + json: async () => ({ success: false, error }), + }) as any)); +} + +/** Run the command in catalog mode; return everything it printed and how it exited. */ +async function runInstall(): Promise<{ out: string; exit: number | undefined }> { + const lines: string[] = []; + const capture = (...args: unknown[]) => { lines.push(args.map(String).join(' ')); }; + vi.spyOn(console, 'log').mockImplementation(capture); + vi.spyOn(console, 'error').mockImplementation(capture); + let exit: number | undefined; + try { + await PackageInstall.run(['com.example.handlerjobs', '--runtime', 'http://runtime.test']); + } catch (err: any) { + exit = err?.oclif?.exit ?? err?.code; + } + return { out: lines.join('\n'), exit }; +} + +const REMEDY = + "Package com.example.handlerjobs was not installed: its enabled job 'tick_job' (handler 'tick') has no `body`, " + + 'so this install door cannot run it. Give the job a `body`, or boot the artifact with `os start --artifact`.'; + +describe('os package install — a refusal is printed with its code', () => { + afterEach(() => { + vi.unstubAllGlobals(); + vi.restoreAllMocks(); + }); + + it('prints status, code and the remedy-carrying message, and exits 1', async () => { + stubRefusal(422, { code: 'VALIDATION_ERROR', message: REMEDY }); + + const { out, exit } = await runInstall(); + + expect(out).toContain(`Install failed (422 VALIDATION_ERROR): ${REMEDY}`); + expect(exit).toBe(1); + }); + + it('renders any code the same way — no case per code', async () => { + stubRefusal(409, { code: 'MANIFEST_CONFLICT', message: 'already defined by local code' }); + + const { out, exit } = await runInstall(); + + expect(out).toContain('Install failed (409 MANIFEST_CONFLICT): already defined by local code'); + expect(exit).toBe(1); + }); + + it('an envelope with no code prints the status alone — never an invented code', async () => { + stubRefusal(500, { message: 'boom' }); + + const { out } = await runInstall(); + + expect(out).toContain('Install failed (500): boom'); + }); +}); diff --git a/packages/cloud-connection/src/marketplace-install-local-jobs.test.ts b/packages/cloud-connection/src/marketplace-install-local-jobs.test.ts new file mode 100644 index 0000000000..23039decb2 --- /dev/null +++ b/packages/cloud-connection/src/marketplace-install-local-jobs.test.ts @@ -0,0 +1,340 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #21489 — an installed package's job BODIES are scheduled, on the install + * route and on the `kernel:ready` rehydrate, through the binder's job half + * (`scheduleAppArtifactJobs`, the call `AppPlugin` makes on `kernel:ready`); and + * the install route REFUSES a package whose enabled job has no `body`. + * + * The defect: a package's jobs were never scheduled by this door, hot or after + * a restart. A job's code was only ever a `handler` — the name of a + * `defineStack({ functions })` entry, which travels in the artifact's runtime + * module, never in the JSON this door installs — so a handler-only package + * installed with a 200 and its job never ran, with nothing saying so. + * + * What this file pins, against the scheduler's state and the engine's writes + * rather than a call: + * + * - install: the body job is handed to `IJobService.schedule`, and a run + * executes the body — its `ctx.api` write lands, as system; + * - rehydrate: a ledger entry written by an earlier process schedules the same + * way when a fresh plugin reaches `kernel:ready`; + * - refusal: a handler-only enabled job answers `422 VALIDATION_ERROR` naming + * the job, its handler and both remedies, and the runtime is left exactly as + * it was found — nothing registered, persisted or scheduled; + * - a package without jobs, and one whose handler-only job is DISABLED, + * install unchanged. + * + * The binder is the REAL `@objectstack/runtime` export (resolved through its + * `exports`, i.e. its built `dist/`, like the plugin's own lazy import), and so + * is the QuickJS sandbox the body runs in. + */ + +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +// The first load of the runtime's dist paid at module top, never inside a +// clocked `it` (the clocked-window rule, `scripts/check-test-source-alias.mjs`): +// the plugin reaches the same module through a dynamic `import()`. +import '@objectstack/runtime'; +import { SCHEDULED_WORK_ENV } from '@objectstack/types'; +import { MarketplaceInstallLocalPlugin } from './marketplace-install-local-plugin.js'; +import { installerAuthService, withInstallerGrants } from './install-local-principal.fixtures.js'; +import { LocalManifestSource } from './local-manifest-source.js'; + +const APP_ID = 'com.example.jobsapp'; +const TICK = 'jobs_app_tick'; +const INTERVAL = { type: 'interval', intervalMs: 1000 }; + +const BODY_JOB = { + name: 'jobs_app_tick_body', + schedule: INTERVAL, + body: { language: 'js', capabilities: ['api.write'], source: `await ctx.api.object('${TICK}').insert({ name: 'tick' });` }, + timeoutMs: 10_000, +}; +const HANDLER_JOB = { name: 'jobs_app_tick_handler', schedule: INTERVAL, handler: 'tick' }; + +/** The compiled-artifact shape `os build` writes and `os package install` sends. */ +function artifact(jobs: unknown[] | undefined, id: string = APP_ID) { + return { + manifest: { id, namespace: 'jobs_app', version: '0.1.0', type: 'app', name: 'Jobs App' }, + objects: [{ name: TICK, label: 'Tick', fields: { name: { type: 'text', label: 'Name' } } }], + ...(jobs ? { jobs } : {}), + }; +} + +/** The engine surface a job body's `ctx.api` writes through, as state. */ +function recordingEngine() { + const writes: Array<{ object: string; data: unknown; context: unknown }> = []; + return { + writes, + engine: { + syncSchemas: async () => undefined, + createContext: (context: unknown) => ({ + object: (object: string) => ({ + insert: async (data: Record) => { + writes.push({ object, data, context }); + return { id: `r${writes.length}`, ...data }; + }, + }), + }), + }, + }; +} + +/** `IJobService` as state: what was handed to `schedule`, by job name. */ +function recordingJobService() { + const scheduled = new Map Promise; options: unknown }>(); + const cancels: string[] = []; + return { + scheduled, + cancels, + svc: { + schedule: vi.fn(async (name: string, _schedule: unknown, run: (c: any) => Promise, options?: unknown) => { + scheduled.set(name, { run, options }); + }), + // The adapters' own semantics: `cancel` stops the job and forgets it. + cancel: async (name: string) => { cancels.push(name); scheduled.delete(name); }, + trigger: async () => undefined, + }, + }; +} + +/** + * The protocol's uninstall-cleanup registry, as its two verbs behave + * (`packages/metadata-protocol`, #21490): one cleanup per name, and ONE runner + * that calls every registered cleanup with the package id and reports each + * outcome. The runner itself is pinned where it lives; this models it so the + * door's `DELETE` reaches what the binder registered. + */ +function registryProtocol() { + const cleanups = new Map Promise<{ success: boolean; removed: number; error?: string }>>(); + return { + registerUninstallCleanup: (name: string, cleanup: any) => { cleanups.set(name, cleanup); }, + runUninstallCleanups: async (request: { packageId: string }) => { + const out: unknown[] = []; + for (const [name, cleanup] of cleanups) out.push({ name, ...(await cleanup({ packageId: request.packageId })) }); + return out; + }, + }; +} + +function makeDeleteC(manifestId: string) { + const json = vi.fn((payload: any, status?: number) => ({ payload, status: status ?? 200 })); + return { + req: { + url: `http://localhost:3000/api/v1/marketplace/install-local/${manifestId}`, + raw: new Request('http://localhost:3000/x'), + json: async () => ({}), + param: (name: string) => (name === 'manifestId' ? manifestId : undefined), + }, + json, + }; +} + +/** + * The package ids handed to `manifest.register` — the plugin also registers its + * own Setup nav bundle at `kernel:ready`, which is not the package under test. + */ +const registered = (register: ReturnType) => + register.mock.calls.map(([m]) => (m as { id?: string })?.id).filter((id) => id === APP_ID); + +type Handler = (c: any) => Promise; + +function makeRawApp() { + const routes = new Map(); + return { + routes, + get: (p: string, h: Handler) => routes.set(`GET ${p}`, h), + post: (p: string, h: Handler) => routes.set(`POST ${p}`, h), + delete: (p: string, h: Handler) => routes.set(`DELETE ${p}`, h), + }; +} + +function makeC(body: any) { + const json = vi.fn((payload: any, status?: number) => ({ payload, status: status ?? 200 })); + return { + req: { + url: 'http://localhost:3000/api/v1/marketplace/install-local', + raw: new Request('http://localhost:3000/x'), + json: async () => body, + param: () => undefined, + }, + json, + }; +} + +let dir: string; +let priorSwitch: string | undefined; +beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'mil-jobs-')); + // [#17396] Package-authored scheduled work is OFF by default in every + // posture; this runtime is one that runs it. + priorSwitch = process.env[SCHEDULED_WORK_ENV]; + process.env[SCHEDULED_WORK_ENV] = 'true'; +}); +afterEach(() => { + rmSync(dir, { recursive: true, force: true }); + if (priorSwitch === undefined) delete process.env[SCHEDULED_WORK_ENV]; + else process.env[SCHEDULED_WORK_ENV] = priorSwitch; + vi.restoreAllMocks(); +}); + +async function bootPlugin() { + const rec = recordingEngine(); + const jobs = recordingJobService(); + const register = vi.fn(); + const rawApp = makeRawApp(); + const hooks = new Map(); + const logger = { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }; + const services: Record = { + manifest: { register }, + auth: installerAuthService(), + objectql: withInstallerGrants(rec.engine), + job: jobs.svc, + protocol: registryProtocol(), + }; + const ctx = { + hook: (e: string, h: any) => hooks.set(e, h), + getService: (name: string) => { + if (name === 'http-server') return { getRawApp: () => rawApp }; + const svc = services[name]; + if (svc === undefined) throw new Error(`no ${name}`); + return svc; + }, + logger, + }; + const plugin = new MarketplaceInstallLocalPlugin({ controlPlaneUrl: 'off', storageDir: dir }); + await plugin.start(ctx as any); + await hooks.get('kernel:ready')?.(); + const install = async (bundle: unknown) => + rawApp.routes.get('POST /api/v1/marketplace/install-local')!(makeC({ manifest: bundle })); + const uninstall = async (manifestId: string) => + rawApp.routes.get('DELETE /api/v1/marketplace/install-local/:manifestId')!(makeDeleteC(manifestId)); + return { install, uninstall, rec, jobs, register, logger }; +} + +describe('#21489: install-local schedules an installed package’s job bodies', () => { + it('install — the body job is scheduled, and a run executes the body (its write lands, as system)', async () => { + const { install, rec, jobs } = await bootPlugin(); + + const res = await install(artifact([BODY_JOB])); + + expect(res.status, JSON.stringify(res.payload)).toBe(200); + expect([...jobs.scheduled.keys()]).toEqual([BODY_JOB.name]); + // The job's own `timeoutMs` reaches the adapter. + expect(jobs.scheduled.get(BODY_JOB.name)!.options).toEqual({ retryPolicy: undefined, timeoutMs: 10_000 }); + + await jobs.scheduled.get(BODY_JOB.name)!.run({ jobId: BODY_JOB.name }); + + expect(rec.writes).toEqual([{ object: TICK, data: { name: 'tick' }, context: { isSystem: true } }]); + }); + + it('rehydrate — a ledger entry from an earlier process schedules its body job at kernel:ready', async () => { + const { manifest: meta, ...sections } = artifact([BODY_JOB]); + new LocalManifestSource(dir).write({ + packageId: APP_ID, + versionId: 'local', + manifestId: APP_ID, + version: '0.1.0', + // What the install route persists: the compiled bundle, flattened. + manifest: { ...meta, ...sections }, + installedAt: '2026-01-01T00:00:00.000Z', + installedBy: 'admin', + withSampleData: false, + }); + + const { rec, jobs } = await bootPlugin(); + + expect([...jobs.scheduled.keys()], 'a restart leaves the installed job unscheduled').toEqual([BODY_JOB.name]); + await jobs.scheduled.get(BODY_JOB.name)!.run({ jobId: BODY_JOB.name }); + expect(rec.writes).toHaveLength(1); + }); +}); + +describe('#21489: an uninstalled or replaced package’s jobs STOP', () => { + const OTHER_ID = 'com.example.otherjobs'; + const OTHER_JOB = { ...BODY_JOB, name: 'other_jobs_tick_body' }; + + it('DELETE cancels the uninstalled package’s jobs through the uninstall cleanup — and no other package’s', async () => { + const { install, uninstall, jobs } = await bootPlugin(); + expect((await install(artifact([BODY_JOB]))).status).toBe(200); + expect((await install(artifact([OTHER_JOB], OTHER_ID))).status).toBe(200); + + const res = await uninstall(APP_ID); + + expect(res.status, JSON.stringify(res.payload)).toBe(200); + expect(jobs.cancels).toEqual([BODY_JOB.name]); + expect([...jobs.scheduled.keys()], 'the control package’s job keeps running').toEqual([OTHER_JOB.name]); + expect(res.payload.data.cleanups).toContainEqual({ name: 'runtime.package-jobs', success: true, removed: 1 }); + }); + + it('a reinstall whose new version DROPS a job cancels it, and keeps the job it still declares', async () => { + const { install, jobs } = await bootPlugin(); + const KEPT = { ...BODY_JOB, name: 'jobs_app_kept' }; + const GONE = { ...BODY_JOB, name: 'jobs_app_gone' }; + expect((await install(artifact([KEPT, GONE]))).status).toBe(200); + + expect((await install(artifact([KEPT]))).status).toBe(200); + + expect(jobs.cancels).toEqual([GONE.name]); + expect([...jobs.scheduled.keys()]).toEqual([KEPT.name]); + }); +}); + +describe('#21489: install-local refuses an enabled job with no body', () => { + it('answers 422 VALIDATION_ERROR naming the job, its handler and both remedies — and changes nothing', async () => { + const { install, rec, jobs, register } = await bootPlugin(); + + const res = await install(artifact([BODY_JOB, HANDLER_JOB])); + + expect(res.status).toBe(422); + expect(res.payload.success).toBe(false); + expect(res.payload.error.code).toBe('VALIDATION_ERROR'); + const message: string = res.payload.error.message; + expect(message).toContain(`'${HANDLER_JOB.name}' (handler 'tick')`); + expect(message).toMatch(/give the job a `body`/i); + expect(message).toContain('os start --artifact'); + // The runtime is left exactly as it was found. + expect(registered(register), 'a refused package must not be registered').toEqual([]); + expect(new LocalManifestSource(dir).read(APP_ID).entry, 'nor persisted').toBeNull(); + expect(jobs.svc.schedule, 'nor any of its jobs scheduled — not even its body job').not.toHaveBeenCalled(); + expect(rec.writes).toEqual([]); + }); + + it('names every refused job, so the author fixes them in one pass', async () => { + const { install } = await bootPlugin(); + const other = { name: 'jobs_app_other', schedule: INTERVAL }; + + const res = await install(artifact([HANDLER_JOB, other])); + + expect(res.status).toBe(422); + expect(res.payload.error.message).toContain(`2 of its enabled jobs '${HANDLER_JOB.name}' (handler 'tick'), 'jobs_app_other' (no handler) have no`); + }); + + it('a DISABLED handler-only job does not block the install, and is not scheduled', async () => { + const { install, jobs, register } = await bootPlugin(); + + const res = await install(artifact([{ ...HANDLER_JOB, enabled: false }])); + + expect(res.status, JSON.stringify(res.payload)).toBe(200); + expect(registered(register)).toEqual([APP_ID]); + expect(jobs.svc.schedule).not.toHaveBeenCalled(); + }); + + it('a package without jobs installs unchanged — the same answer, nothing scheduled', async () => { + const { install, jobs, register } = await bootPlugin(); + + const res = await install(artifact(undefined)); + + expect(res.status, JSON.stringify(res.payload)).toBe(200); + expect(res.payload.success).toBe(true); + expect(Object.keys(res.payload.data).sort()).toEqual([ + 'hotLoaded', 'installedAt', 'manifestId', 'note', 'seeded', 'storageDir', + 'translationsLoaded', 'upgradedFrom', 'version', 'versionId', + ]); + expect(registered(register)).toEqual([APP_ID]); + expect(jobs.svc.schedule).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/cloud-connection/src/marketplace-install-local-plugin.ts b/packages/cloud-connection/src/marketplace-install-local-plugin.ts index 7667b86205..410bc0b938 100644 --- a/packages/cloud-connection/src/marketplace-install-local-plugin.ts +++ b/packages/cloud-connection/src/marketplace-install-local-plugin.ts @@ -27,6 +27,10 @@ * before anything else looks at it; an id the declaration refuses * answers `PLUGIN_MANIFEST_INVALID` (400 for an inline manifest, * 502 for a cloud snapshot) and nothing is registered or written. + * A package declaring an enabled job with no `body` answers + * `VALIDATION_ERROR` (422) the same way: a job's function-name + * `handler` is code no JSON door carries. The package's job bodies + * are scheduled on install and on every rehydrate. * * GET /api/v1/marketplace/install-local * → lists currently installed marketplace packages. Requires an @@ -157,6 +161,47 @@ const REGISTRY_WITHDRAWAL = 'registry.uninstallPackage'; */ const INSTALL_LOCAL_CAPABILITY = 'manage_metadata'; +/** + * [#21489] The refusal of a package that declares an enabled job no JSON door + * can run: `VALIDATION_ERROR` / 422. + * + * The code is the standard catalog's input-validation member. The condition is + * that the install payload fails this door's acceptance rule — every enabled + * job carries a `body` — and the ledger's admission rule sends a generic + * validation condition to the standard member rather than to a registered + * synonym (`error-code-ledger.zod.ts`, "Registering a new code"). What the + * author does instead is the prescription the message carries. + * `PLUGIN_MANIFEST_INVALID` is deliberately not it: the manifest is valid — + * `os validate` passes it and `os start --artifact` runs it. + * + * The status is 422, not the 400 / 502 split this door uses for an invalid + * manifest: the package is well-formed and this door cannot process it, which + * holds whichever branch supplied it — a catalog package declaring a handler job + * is no upstream fault. 422 derives `VALIDATION_ERROR` + * (`standardErrorCodeForHttpStatus`), so code and status agree. + */ +const JOB_WITHOUT_BODY_REFUSAL_CODE = 'VALIDATION_ERROR'; +const JOB_WITHOUT_BODY_REFUSAL_STATUS = 422; + +/** + * The refusal sentence: which jobs, why this door cannot run them, and the two + * remedies — a `body` (every door), or an `--artifact` boot (which loads the + * runtime module a `handler` lives in). Names each job and the function its + * `handler` declares, so the author fixes them all in one pass. + */ +function describeJobsWithoutBody(manifestId: string, jobs: ReadonlyArray<{ name: string; handler?: string }>): string { + const list = jobs + .map((j) => `'${j.name}' (${j.handler !== undefined ? `handler '${j.handler}'` : 'no handler'})`) + .join(', '); + const one = jobs.length === 1; + return `Package ${manifestId} was not installed: ${one ? 'its enabled job' : `${jobs.length} of its enabled jobs`} ` + + `${list} ${one ? 'has' : 'have'} no \`body\`, so this install door cannot run ${one ? 'it' : 'them'}. ` + + "A job's `handler` names a `defineStack({ functions })` entry, which is code: it travels in the artifact's " + + 'runtime module, never in the package JSON this door installs, so the job would be installed and never run. ' + + 'Give the job a `body` (sandboxed JS, the form hooks and actions use, which travels with the package and runs ' + + 'on every door), or boot the artifact with `os start --artifact`, which loads its runtime module.'; +} + /** * A ledger read failure in the thrower's own words (#5413 / #5426). * @@ -863,6 +908,32 @@ export class MarketplaceInstallLocalPlugin implements Plugin { const manifestId = declaredId.data; if (inlineManifest) packageId = manifestId; + // 1c. [#21489] ⭐ A JOB THIS DOOR CANNOT RUN IS REFUSED, not installed. + // A job runs on a JSON door only through its sandboxed `body`; its + // deprecated `handler` names a `defineStack({ functions })` entry, + // which is code and travels only in the artifact's runtime module — + // never in the package this door receives. Before this, such a + // package installed with a 200 and its job was declared and never + // scheduled, hot or after a restart, with nothing anywhere saying so. + // + // Answered here, beside the id gate and ahead of the collision check, + // the posture gate, the hot-register and the ledger write, so a + // refused install leaves the runtime exactly as it found it and the + // author can add the body and retry. Only an ENABLED job is judged: + // a disabled one is never scheduled on any door. ⛔ Rehydrate is not + // gated, for the id gate's reason: an entry an older build installed + // still rehydrates (its handler-only job is reported, not run). + const withoutBody = await this.jobsWithoutBody(ctx, manifest, manifestId); + if (withoutBody.length > 0) { + return c.json({ + success: false, + error: { + code: JOB_WITHOUT_BODY_REFUSAL_CODE, + message: describeJobsWithoutBody(manifestId, withoutBody), + }, + }, JOB_WITHOUT_BODY_REFUSAL_STATUS); + } + // 2. Conflict check — refuse to overwrite user-authored apps const conflict = this.findConflict(ctx, manifestId); if (conflict === 'user-code') { @@ -1621,6 +1692,13 @@ export class MarketplaceInstallLocalPlugin implements Plugin { * (`app:`). Called on the install route and on the * `kernel:ready` rehydrate. * + * [#21489] …and schedule its jobs through `scheduleAppArtifactJobs`, the + * binder's job half and the call `AppPlugin` makes on `kernel:ready`: a job + * `body` runs sandboxed on its schedule, hot and after a restart. A + * handler-only job never reaches here on an install — the install route + * refuses it ({@link jobsWithoutBody}) — and on a rehydrate of an entry an + * older build installed it is reported at `warn` and not run. + * * Before this, `manifest.register` was the whole install: the package's * actions and hooks were DECLARED and never bound, so every door refused * its script actions ("No handler registered" over MCP, 404 over REST) @@ -1632,7 +1710,8 @@ export class MarketplaceInstallLocalPlugin implements Plugin { * dropped. ⛔ No second registration path lives here: a runtime without the * binder (an older build, or a suite that mocks `@objectstack/runtime` * without it) binds NOTHING and says so — the package's script actions then - * stay unrunnable, and `list_actions` does not advertise them. + * stay unrunnable, and `list_actions` does not advertise them. The same + * holds for the job half: no job is scheduled here by any other route. * * Resolved lazily through `@objectstack/runtime`, like every other runtime * helper this plugin calls. Never throws. @@ -1641,22 +1720,62 @@ export class MarketplaceInstallLocalPlugin implements Plugin { let ql: IObjectQLEngine | undefined; try { ql = ctx.getService('objectql'); } catch { /* no data engine */ } if (!ql) { - ctx.logger?.warn?.(`[MarketplaceInstallLocal] no objectql engine — the script actions and body hooks of ${manifestId} are NOT bound`); + ctx.logger?.warn?.(`[MarketplaceInstallLocal] no objectql engine — the script actions, body hooks and jobs of ${manifestId} are NOT bound`); return; } let bind: typeof import('@objectstack/runtime')['bindAppArtifactHandlers'] | undefined; + let schedule: typeof import('@objectstack/runtime')['scheduleAppArtifactJobs'] | undefined; try { const mod: any = await import('@objectstack/runtime'); if (typeof mod?.bindAppArtifactHandlers === 'function') bind = mod.bindAppArtifactHandlers; + if (typeof mod?.scheduleAppArtifactJobs === 'function') schedule = mod.scheduleAppArtifactJobs; } catch { /* reported below */ } if (!bind) { ctx.logger?.warn?.( `[MarketplaceInstallLocal] this runtime has no bindAppArtifactHandlers — the script actions and body hooks of ${manifestId} are NOT bound: ` + 'every door refuses those actions and the hooks never fire. Upgrade @objectstack/runtime alongside @objectstack/cloud-connection.', ); + } else { + bind(ql, manifest, { appId: manifestId, logger: ctx.logger, source: 'MarketplaceInstallLocal' }); + } + if (!schedule) { + ctx.logger?.warn?.( + `[MarketplaceInstallLocal] this runtime has no scheduleAppArtifactJobs — the jobs of ${manifestId} are NOT scheduled. ` + + 'Upgrade @objectstack/runtime alongside @objectstack/cloud-connection.', + ); return; } - bind(ql, manifest, { appId: manifestId, logger: ctx.logger, source: 'MarketplaceInstallLocal' }); + await schedule(ctx, manifest, { appId: manifestId, ql, source: 'MarketplaceInstallLocal' }); + }; + + /** + * [#21489] The enabled jobs of `manifest` that carry no `body` — the jobs + * no JSON door can run, which the install route refuses. The judgement is + * the runtime binder's own (`collectJobsWithoutBody`, which reads the jobs + * the binder schedules), so the door and the binder cannot disagree about + * what this door can run. + * + * A runtime that predates the judgement judges nothing and says so: the + * install proceeds as it did before this gate existed. + */ + private jobsWithoutBody = async ( + ctx: PluginContext, + manifest: unknown, + manifestId: string, + ): Promise> => { + let collect: typeof import('@objectstack/runtime')['collectJobsWithoutBody'] | undefined; + try { + const mod: any = await import('@objectstack/runtime'); + if (typeof mod?.collectJobsWithoutBody === 'function') collect = mod.collectJobsWithoutBody; + } catch { /* reported below */ } + if (!collect) { + ctx.logger?.warn?.( + `[MarketplaceInstallLocal] this runtime has no collectJobsWithoutBody — the jobs of ${manifestId} are not judged, ` + + 'so a job with no `body` installs and is never run. Upgrade @objectstack/runtime alongside @objectstack/cloud-connection.', + ); + return []; + } + return collect(manifest); }; /** diff --git a/packages/runtime/src/app-artifact-handlers.jobs.test.ts b/packages/runtime/src/app-artifact-handlers.jobs.test.ts new file mode 100644 index 0000000000..4a8988e400 --- /dev/null +++ b/packages/runtime/src/app-artifact-handlers.jobs.test.ts @@ -0,0 +1,440 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #21489 — the binder's job half: `scheduleAppArtifactJobs`, the ONE place a + * declared job becomes a scheduled one, and `collectJobsWithoutBody`, the + * judgement the install-local door refuses on. + * + * Its callers are pinned where they live (`AppPlugin` below and in + * `app-plugin.jobs.test.ts`; the install-local plugin in + * `@objectstack/cloud-connection`; the public door in + * `packages/cli/test/package-install-local-jobs.integration.test.ts`). This file + * pins the contract they share, against the REAL QuickJS sandbox: + * + * - a job `body` is scheduled, and a run executes the body — its `ctx.api` + * write reaches the engine, as system; + * - with both keys present the `body` wins, and a body that cannot be bound + * schedules NOTHING (never the handler beside it); + * - a `handler` job still runs its `functions` entry, and one with no entry is + * not scheduled, with the remedy said; + * - the job's `timeoutMs` is the body's one limit; with none, the runner's + * JOB default applies — not the hook's, not the action's; + * - the body's return is read as a `JobRunOutcome`, in that shape only; + * - a body's `ctx` carries no job name and no trigger data. + */ + +import { describe, it, expect, vi } from 'vitest'; +import type { PluginContext } from '@objectstack/core'; +import { scheduleAppArtifactJobs, collectJobsWithoutBody, PACKAGE_JOBS_UNINSTALL_CLEANUP } from './app-artifact-handlers.js'; +import { jobBodyRunnerFactory } from './sandbox/body-runner.js'; +import { QuickJSScriptRunner } from './sandbox/quickjs-runner.js'; +import { AppPlugin } from './app-plugin.js'; +import { withScheduledWorkOn } from './scheduled-work.test-support.js'; + +// [#17396] Package-authored jobs are scheduled only where the deployment runs +// package scheduled work, OFF by default; these suites measure the binder. +withScheduledWorkOn(); + +const APP_ID = 'com.example.jobsapp'; +const TICK = 'jobs_app_tick'; +const INTERVAL = { type: 'interval', intervalMs: 1000 }; + +/** A body that writes one row per run. */ +const WRITE_BODY = { + language: 'js', + capabilities: ['api.write'], + source: `await ctx.api.object('${TICK}').insert({ name: 'tick' });`, +}; + +/** The engine surface a job body's `ctx.api` reaches, as state: every write and the envelope it ran under. */ +function recordingEngine() { + const writes: Array<{ object: string; data: unknown; context: unknown }> = []; + return { + writes, + ql: { + createContext: (context: unknown) => ({ + object: (object: string) => ({ + insert: async (data: Record) => { + writes.push({ object, data, context }); + return { id: `r${writes.length}`, ...data }; + }, + }), + }), + }, + }; +} + +/** + * `IJobService` as state: what is scheduled right now, by job name — `schedule` + * replaces by name and `cancel` removes, the adapters' own semantics — plus + * every cancel in order. `failCancel` names a job whose cancel throws. + */ +function recordingJobService(opts: { failCancel?: string } = {}) { + const scheduled = new Map Promise; options: unknown }>(); + const cancels: string[] = []; + return { + scheduled, + cancels, + svc: { + schedule: async (name: string, schedule: unknown, run: (c: any) => Promise, options?: unknown) => { + scheduled.set(name, { schedule, run, options }); + }, + cancel: async (name: string) => { + if (name === opts.failCancel) throw new Error(`cannot cancel ${name}`); + cancels.push(name); + scheduled.delete(name); + }, + trigger: async () => undefined, + }, + }; +} + +/** The protocol's uninstall-cleanup registry as state — the two verbs the binder and the doors use. */ +function recordingProtocol() { + const cleanups = new Map Promise<{ success: boolean; removed: number; error?: string }>>(); + const registrations: string[] = []; + return { + cleanups, + registrations, + protocol: { + registerUninstallCleanup: (name: string, cleanup: any) => { registrations.push(name); cleanups.set(name, cleanup); }, + }, + }; +} + +function harness(opts: { failCancel?: string; withProtocol?: boolean } = {}) { + const engine = recordingEngine(); + const jobs = recordingJobService({ failCancel: opts.failCancel }); + const reg = recordingProtocol(); + const logger = { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() }; + const ctx = { + logger, + getService: (name: string) => { + if (name === 'job') return jobs.svc; + if (name === 'protocol' && opts.withProtocol) return reg.protocol; + throw new Error(`no ${name}`); + }, + } as unknown as PluginContext; + const schedule = (bundle: unknown, appId: string = APP_ID) => + scheduleAppArtifactJobs(ctx, bundle, { appId, ql: engine.ql as any, source: 'Test' }); + const warned = () => logger.warn.mock.calls.map((c) => String(c[0])); + const errored = () => logger.error.mock.calls.map((c) => String(c[0])); + return { engine, jobs, reg, logger, ctx, schedule, warned, errored }; +} + +/** A flattened JSON package, as install-local holds it: no `functions`. */ +const pkg = (jobs: unknown[], extra: Record = {}) => ({ + id: APP_ID, version: '0.1.0', type: 'app', jobs, ...extra, +}); + +describe('#21489: scheduleAppArtifactJobs — job bodies', () => { + it('schedules a body job, and a run executes the body — its ctx.api write reaches the engine as system', async () => { + const h = harness(); + + const out = await h.schedule(pkg([{ name: 'tick_job', schedule: INTERVAL, body: WRITE_BODY }])); + + expect(out.bodies).toEqual(['tick_job']); + expect(out.notScheduled).toEqual([]); + expect([...h.jobs.scheduled.keys()]).toEqual(['tick_job']); + // The authored schedule, lowered to the boundary tier. + expect(h.jobs.scheduled.get('tick_job')!.schedule).toEqual({ type: 'interval', intervalMs: 1000 }); + + await h.jobs.scheduled.get('tick_job')!.run({ jobId: 'tick_job' }); + + expect(h.engine.writes).toEqual([{ object: TICK, data: { name: 'tick' }, context: { isSystem: true } }]); + }); + + it('with both keys present the body WINS — the handler beside it is never called', async () => { + const h = harness(); + const tick = vi.fn(async () => undefined); + + const out = await h.schedule(pkg( + [{ name: 'both_job', schedule: INTERVAL, body: WRITE_BODY, handler: 'tick' }], + { functions: { tick } }, + )); + await h.jobs.scheduled.get('both_job')!.run({ jobId: 'both_job' }); + + expect(out.bodies).toEqual(['both_job']); + expect(out.handlers).toEqual([]); + expect(tick).not.toHaveBeenCalled(); + expect(h.engine.writes).toHaveLength(1); + }); + + it('an expression (L1) body is NOT scheduled — and the handler beside it is not run instead', async () => { + const h = harness(); + const tick = vi.fn(async () => undefined); + + const out = await h.schedule(pkg( + [{ name: 'l1_job', schedule: INTERVAL, body: { language: 'expression', source: '1 + 1' }, handler: 'tick' }], + { functions: { tick } }, + )); + + expect(out.notScheduled).toEqual(['l1_job']); + expect(h.jobs.scheduled.size).toBe(0); + expect(h.warned().some((m) => m.includes('invalid job.body shape'))).toBe(true); + }); + + it('a body.timeoutMs (refused on a job by the spec) is NOT run under a second limit — the job is not scheduled', async () => { + const h = harness(); + + const out = await h.schedule(pkg([{ name: 'two_limits', schedule: INTERVAL, body: { ...WRITE_BODY, timeoutMs: 100 } }])); + + expect(out.notScheduled).toEqual(['two_limits']); + expect(h.jobs.scheduled.size).toBe(0); + expect(h.warned().some((m) => m.includes('`body.timeoutMs`') && m.includes("job's own `timeoutMs`"))).toBe(true); + }); + + it("the job's timeoutMs is threaded to the adapter AND bounds the sandbox run (the one limit)", async () => { + const h = harness(); + + await h.schedule(pkg([{ + name: 'spin_job', + schedule: INTERVAL, + timeoutMs: 40, + body: { language: 'js', source: 'while (true) {}' }, + }])); + const entry = h.jobs.scheduled.get('spin_job')!; + + expect(entry.options).toEqual({ retryPolicy: undefined, timeoutMs: 40 }); + await expect(entry.run({ jobId: 'spin_job' })).rejects.toThrow(/job 'spin_job' exceeded CPU budget of 40ms/); + }); + + it('a body returns its JobRunOutcome — copied in the declared shape only', async () => { + const h = harness(); + await h.schedule(pkg([ + { name: 'degraded_job', schedule: INTERVAL, body: { language: 'js', source: "return { outcome: 'degraded', reason: 'nothing to sweep', extra: 1 };" } }, + { name: 'number_job', schedule: INTERVAL, body: { language: 'js', source: 'return 5;' } }, + ])); + + await expect(h.jobs.scheduled.get('degraded_job')!.run({ jobId: 'degraded_job' })) + .resolves.toEqual({ outcome: 'degraded', reason: 'nothing to sweep' }); + await expect(h.jobs.scheduled.get('number_job')!.run({ jobId: 'number_job' })).resolves.toBeUndefined(); + }); + + it("a body's ctx carries no job name and no trigger data — api / log / crypto is its surface", async () => { + const h = harness(); + await h.schedule(pkg([{ + name: 'probe_job', + schedule: INTERVAL, + body: { + language: 'js', + capabilities: ['api.read', 'log', 'crypto.uuid'], + source: "return { outcome: 'completed', reason: JSON.stringify({ jobId: typeof ctx.jobId, data: typeof ctx.data, " + + "api: typeof ctx.api.object, log: typeof ctx.log.info, uuid: typeof ctx.crypto.randomUUID }) };", + }, + }])); + + // A manual trigger hands `data`; the body still does not receive it. + const out = await h.jobs.scheduled.get('probe_job')!.run({ jobId: 'probe_job', data: { a: 1 } }) as { reason: string }; + + expect(JSON.parse(out.reason)).toEqual({ + jobId: 'undefined', data: 'undefined', api: 'function', log: 'function', uuid: 'function', + }); + }); +}); + +describe('#21489: scheduleAppArtifactJobs — handler jobs and the door-wide gates', () => { + it('a handler job still runs its functions entry, with the in-process JobHandlerContext (control)', async () => { + const h = harness(); + let seen: any; + const tick = vi.fn(async (c: any) => { seen = c; }); + + const out = await h.schedule(pkg([{ name: 'handler_job', schedule: INTERVAL, handler: 'tick' }], { functions: { tick } })); + await h.jobs.scheduled.get('handler_job')!.run({ jobId: 'handler_job' }); + + expect(out.handlers).toEqual(['handler_job']); + expect(tick).toHaveBeenCalledTimes(1); + expect(seen.jobId).toBe('handler_job'); + expect(seen.ql).toBe(h.engine.ql); + }); + + it('a handler job with no functions entry (a JSON package) is NOT scheduled, and the warn names the body remedy', async () => { + const h = harness(); + + const out = await h.schedule(pkg([{ name: 'handler_job', schedule: INTERVAL, handler: 'tick' }])); + + expect(out.notScheduled).toEqual(['handler_job']); + expect(h.jobs.scheduled.size).toBe(0); + expect(h.warned().some((m) => m.includes('job handler not found') && m.includes('give the job a `body`'))).toBe(true); + }); + + it('a disabled job is skipped on every form', async () => { + const h = harness(); + + const out = await h.schedule(pkg([ + { name: 'off_body', schedule: INTERVAL, body: WRITE_BODY, enabled: false }, + { name: 'off_handler', schedule: INTERVAL, handler: 'tick', enabled: false }, + ])); + + expect(out).toEqual({ bodies: [], handlers: [], notScheduled: [], failed: [], cancelled: [] }); + expect(h.jobs.scheduled.size).toBe(0); + }); + + it('the deployment switch OFF withholds every job, said once', async () => { + const h = harness(); + const prior = process.env.OS_AUTOMATION_SCHEDULED_WORK_ENABLED; + delete process.env.OS_AUTOMATION_SCHEDULED_WORK_ENABLED; + try { + const out = await h.schedule(pkg([{ name: 'tick_job', schedule: INTERVAL, body: WRITE_BODY }])); + expect(out.withheld).toBe('scheduled-work-disabled'); + } finally { + process.env.OS_AUTOMATION_SCHEDULED_WORK_ENABLED = prior; + } + expect(h.jobs.scheduled.size).toBe(0); + }); +}); + +describe('#21489: re-scheduling replaces — a job the new version does not schedule is CANCELLED', () => { + const job = (name: string) => ({ name, schedule: INTERVAL, body: WRITE_BODY }); + + it('a reinstall that DROPS a job cancels it, and keeps the one it still declares', async () => { + const h = harness(); + await h.schedule(pkg([job('kept_job'), job('gone_job')])); + + const out = await h.schedule(pkg([job('kept_job')])); + + expect(out.cancelled).toEqual(['gone_job']); + expect(h.jobs.cancels).toEqual(['gone_job']); + expect([...h.jobs.scheduled.keys()]).toEqual(['kept_job']); + }); + + it('a version that disables a job, or declares no jobs at all, cancels what it no longer runs', async () => { + const h = harness(); + await h.schedule(pkg([job('a_job'), job('b_job')])); + + const disabled = await h.schedule(pkg([job('a_job'), { ...job('b_job'), enabled: false }])); + expect(disabled.cancelled).toEqual(['b_job']); + + const none = await h.schedule(pkg([])); + expect(none.cancelled).toEqual(['a_job']); + expect(h.jobs.scheduled.size).toBe(0); + }); + + it("another app's jobs are never cancelled — not even one that took over a name this app once scheduled", async () => { + const h = harness(); + await h.schedule(pkg([job('shared_name'), job('mine_only')]), APP_ID); + await h.schedule(pkg([job('shared_name'), job('theirs_only')]), 'com.example.other'); + + // This app's next version drops both: only its own remaining job stops. + const out = await h.schedule(pkg([]), APP_ID); + + expect(out.cancelled).toEqual(['mine_only']); + expect([...h.jobs.scheduled.keys()].sort()).toEqual(['shared_name', 'theirs_only']); + }); + + it('a cancel that throws is said at error, and the job stays on the record for the next attempt', async () => { + const h = harness({ failCancel: 'stuck_job' }); + await h.schedule(pkg([job('stuck_job')])); + + const first = await h.schedule(pkg([])); + expect(first.cancelled).toEqual([]); + expect(h.errored().some((m) => m.includes('could NOT be cancelled'))).toBe(true); + }); +}); + +describe('#21489: the uninstall cleanup cancels the uninstalled package\'s jobs', () => { + const job = (name: string) => ({ name, schedule: INTERVAL, body: WRITE_BODY }); + + it('is registered once per protocol, as runtime.package-jobs, when a package\'s jobs are scheduled', async () => { + const h = harness({ withProtocol: true }); + + await h.schedule(pkg([job('a_job')]), APP_ID); + await h.schedule(pkg([job('b_job')]), 'com.example.other'); + + expect(h.reg.registrations).toEqual([PACKAGE_JOBS_UNINSTALL_CLEANUP]); + expect(PACKAGE_JOBS_UNINSTALL_CLEANUP).toBe('runtime.package-jobs'); + }); + + it("cancels every job of the uninstalled package and none of another package's", async () => { + const h = harness({ withProtocol: true }); + await h.schedule(pkg([job('a_job'), job('a_other')]), APP_ID); + await h.schedule(pkg([job('b_job')]), 'com.example.other'); + const cleanup = h.reg.cleanups.get(PACKAGE_JOBS_UNINSTALL_CLEANUP)!; + + const result = await cleanup({ packageId: APP_ID }); + + expect(result).toEqual({ success: true, removed: 2 }); + expect(h.jobs.cancels.sort()).toEqual(['a_job', 'a_other']); + expect([...h.jobs.scheduled.keys()]).toEqual(['b_job']); + // A second uninstall of the same package has nothing left to cancel. + await expect(cleanup({ packageId: APP_ID })).resolves.toEqual({ success: true, removed: 0 }); + }); + + it('a package that scheduled nothing is a no-op', async () => { + const h = harness({ withProtocol: true }); + await h.schedule(pkg([job('a_job')]), APP_ID); + + await expect(h.reg.cleanups.get(PACKAGE_JOBS_UNINSTALL_CLEANUP)!({ packageId: 'com.example.never' })) + .resolves.toEqual({ success: true, removed: 0 }); + expect(h.jobs.cancels).toEqual([]); + }); + + it('a job it could not cancel is an outcome, never a throw — success:false naming the job', async () => { + const h = harness({ withProtocol: true, failCancel: 'stuck_job' }); + await h.schedule(pkg([job('stuck_job'), job('fine_job')]), APP_ID); + + const result = await h.reg.cleanups.get(PACKAGE_JOBS_UNINSTALL_CLEANUP)!({ packageId: APP_ID }); + + expect(result.success).toBe(false); + expect(result.removed).toBe(1); + expect(result.error).toContain('stuck_job'); + }); +}); + +describe('#21489: the sandbox job origin', () => { + it("a job body with no timeoutMs gets the runner's JOB default — not the hook's, not the action's", async () => { + const runner = new QuickJSScriptRunner({ jobTimeoutMs: 30, hookTimeoutMs: 5_000, actionTimeoutMs: 5_000 }); + const bind = jobBodyRunnerFactory(runner, { ql: recordingEngine().ql, appId: APP_ID }); + const run = bind({ name: 'spin_default', body: { language: 'js', source: 'while (true) {}' } })!; + + await expect(run({ jobId: 'spin_default' })).rejects.toThrow(/job 'spin_default' exceeded CPU budget of 30ms/); + }); +}); + +describe('#21489: collectJobsWithoutBody — what no JSON door can run', () => { + it('names each ENABLED job without a body, with the function its handler declares', () => { + expect(collectJobsWithoutBody(pkg([ + { name: 'handler_only', schedule: INTERVAL, handler: 'tick' }, + { name: 'neither', schedule: INTERVAL }, + { name: 'body_job', schedule: INTERVAL, body: WRITE_BODY }, + { name: 'both', schedule: INTERVAL, body: WRITE_BODY, handler: 'tick' }, + { name: 'disabled_handler', schedule: INTERVAL, handler: 'tick', enabled: false }, + ]))).toEqual([ + { name: 'handler_only', handler: 'tick' }, + { name: 'neither' }, + ]); + }); + + it('a package without jobs has nothing to refuse', () => { + expect(collectJobsWithoutBody(pkg([]))).toEqual([]); + expect(collectJobsWithoutBody({ id: APP_ID })).toEqual([]); + }); +}); + +describe('#21489: AppPlugin (the boot door) schedules a body job through the binder', () => { + it('a body-only job — skipped at warn before — is scheduled on kernel:ready, and runs its body', async () => { + const engine = recordingEngine(); + const jobs = recordingJobService(); + const ready: Array<() => Promise> = []; + const ctx = { + logger: { info: vi.fn(), error: vi.fn(), warn: vi.fn(), debug: vi.fn() }, + registerService: vi.fn(), + getService: vi.fn((name: string) => { + if (name === 'job') return jobs.svc; + if (name === 'objectql') return engine.ql; + return undefined; + }), + getServices: vi.fn(() => []), + hook: vi.fn((event: string, cb: () => Promise) => { if (event === 'kernel:ready') ready.push(cb); }), + trigger: vi.fn(), + } as unknown as PluginContext; + const plugin = new AppPlugin({ id: APP_ID, jobs: [{ name: 'boot_body', schedule: INTERVAL, body: WRITE_BODY }] }); + + await plugin.start!(ctx); + for (const cb of ready) await cb(); + await jobs.scheduled.get('boot_body')?.run({ jobId: 'boot_body' }); + + expect([...jobs.scheduled.keys()]).toEqual(['boot_body']); + expect(engine.writes).toEqual([{ object: TICK, data: { name: 'tick' }, context: { isSystem: true } }]); + }); +}); diff --git a/packages/runtime/src/app-artifact-handlers.ts b/packages/runtime/src/app-artifact-handlers.ts index 1f15dc45ab..903c0ea6c5 100644 --- a/packages/runtime/src/app-artifact-handlers.ts +++ b/packages/runtime/src/app-artifact-handlers.ts @@ -30,6 +30,8 @@ * - `AppPlugin.start` (the boot artifact, `defineStack` configs), * - the install-local plugin's install route and its `kernel:ready` rehydrate. * + * (Jobs: the same doors, through this module's job half — see the last section.) + * * ## Re-binding replaces, it never accumulates * * Before binding, the owner's previous set is torn down: its action handlers @@ -49,13 +51,45 @@ * Never throws. A hook set or an action set that fails to bind is logged at * `error` with the app id and the other half still binds; a single action whose * registration throws is logged at `warn` and the rest still register. + * + * ## The job half (#21489) + * + * A job is the third kind of server-side code an artifact declares, and since + * `JobSchema.body` it too can be DATA: a sandboxed body, the hook body shape. + * {@link scheduleAppArtifactJobs} is this binder's job half — the ONE place a + * declared job becomes a scheduled one — and every door above calls it: + * `AppPlugin` on `kernel:ready`, the install-local plugin on its install route + * and its rehydrate. It is a second entry point rather than a fourth block in + * {@link bindAppArtifactHandlers} for one reason, timing: the boot binds hooks + * and actions in `start()`, but schedules jobs only once the kernel is ready + * (the job service and the engine have registered), while install-local's + * doors are already past that point. One implementation, two moments. + * + * A job runs on a JSON door only through its `body`. Its deprecated `handler` + * names a `defineStack({ functions })` entry, which is code: it travels in the + * artifact's runtime module, which only `os start --artifact` loads, so no JSON + * door can ever resolve it. {@link collectJobsWithoutBody} names those jobs, and + * the install-local install route refuses a package that declares one enabled. */ -import type { IObjectQLEngine, Logger } from '@objectstack/spec/contracts'; +import type { PluginContext } from '@objectstack/core'; +import type { IJobService, IObjectQLEngine, JobHandler, Logger } from '@objectstack/spec/contracts'; +import { resolveScheduledWorkEnabled, SCHEDULED_WORK_DISABLED_REASON } from '@objectstack/types'; +import { SEMCONV } from '@objectstack/observability'; import { QuickJSScriptRunner } from './sandbox/quickjs-runner.js'; -import { hookBodyRunnerFactory, actionBodyRunnerFactory } from './sandbox/body-runner.js'; +import { hookBodyRunnerFactory, actionBodyRunnerFactory, jobBodyRunnerFactory } from './sandbox/body-runner.js'; import { GLOBAL_ACTION_OBJECT_KEY } from './action-execution.js'; -import { collectBundleActions, collectBundleFunctionEntries, collectBundleHooks } from './app-plugin.js'; +import { + collectBundleActions, + collectBundleFunctionEntries, + collectBundleFunctions, + collectBundleHooks, + collectBundleJobs, +} from './app-plugin.js'; +import { resolveArtifactCollections } from './artifact-collections.js'; +import { toBoundaryJobSchedule } from './job-schedule.js'; +import type { JobHandlerContext } from './job-handler-context.js'; +import { resolveMetrics } from './observability/observability-service-plugin.js'; /** * The engine owner key every handler bound for `appId` is registered under — @@ -200,3 +234,423 @@ export function bindAppArtifactHandlers( return out; } + +// ─── The job half (#21489) ───────────────────────────────────────────── + +/** + * An enabled job a JSON door cannot run: it carries no `body`. Its `handler` + * (deprecated) names a `defineStack({ functions })` entry — code, which a JSON + * artifact never carries (ADR-0088) — or it names nothing at all. + */ +export interface JobWithoutBody { + /** The job's `name`. */ + name: string; + /** The function name the job's `handler` declares, when it declares one. */ + handler?: string; +} + +/** + * The enabled jobs of an artifact that carry no `body` — the jobs no JSON door + * can schedule (#21489). The install-local install route refuses a package + * that declares one; see the module header. + * + * Reads the jobs the binder reads ({@link collectBundleJobs}), and calls a job + * enabled exactly when the binder does: `enabled: false` is the one value that + * disables it (the schema's default is `true`). + */ +export function collectJobsWithoutBody(bundle: unknown): JobWithoutBody[] { + const out: JobWithoutBody[] = []; + for (const job of collectBundleJobs(bundle)) { + if (!job || typeof job !== 'object') continue; + if (job.enabled === false) continue; + if (job.body) continue; + out.push({ + name: typeof job.name === 'string' ? job.name : String(job.name), + ...(typeof job.handler === 'string' ? { handler: job.handler } : {}), + }); + } + return out; +} + +export interface AppArtifactJobSchedulingOptions { + /** The app the jobs belong to: the artifact manifest's `id` (falling back to `name`). */ + appId: string; + /** + * The engine a job runs against: a body's `ctx.api` is served from it, and a + * `handler` job's in-process context carries it as `ql`. + */ + ql: IObjectQLEngine | undefined; + /** Who is scheduling, for the log lines — `'AppPlugin'`, `'MarketplaceInstallLocal'`. */ + source?: string; +} + +/** What one {@link scheduleAppArtifactJobs} call scheduled. */ +export interface AppArtifactJobScheduling { + /** + * Set when nothing was scheduled for a reason that holds for every job: + * the deployment does not run package-authored scheduled work (#17396), or + * no job service is registered. + */ + withheld?: 'scheduled-work-disabled' | 'no-job-service'; + /** Jobs scheduled to run their sandboxed `body`. */ + bodies: string[]; + /** Jobs scheduled to run the `functions` entry their `handler` names. */ + handlers: string[]; + /** Enabled jobs with nothing this door could run (an unbindable body, an unresolvable handler, no name). */ + notScheduled: string[]; + /** Jobs whose `IJobService.schedule` call threw. */ + failed: string[]; + /** + * Jobs this app had scheduled on the job service before, which this call + * did not schedule again and therefore CANCELLED (#21489): a job the new + * version dropped, disabled or can no longer run. Re-scheduling replaces; + * it never leaves the old set running beside the new one. + */ + cancelled: string[]; +} + +/** + * Schedule every job an app artifact declares onto the running job service — + * the binder's job half (#21489), and the ONE place a declared job becomes a + * scheduled one. Called by `AppPlugin` on `kernel:ready` and by the + * install-local plugin on its install route and its rehydrate. + * + * Per job, in this order: + * + * - `enabled: false` → skipped (debug); + * - a `body` → the sandboxed body (`jobBodyRunnerFactory`), and the `body` + * WINS when a `handler` is declared beside it. A body that cannot be bound + * (wrong shape, a `body.timeoutMs`) schedules nothing — never the handler + * beside it, which would run code the author replaced; + * - else a `handler` → the `functions` entry it names, invoked with the + * in-process `JobHandlerContext` (#14094). A JSON artifact carries no + * functions, so on install-local this resolves nothing — which is why that + * door refuses the shape up front ({@link collectJobsWithoutBody}); + * - else → not scheduled (warn). + * + * The schedule is lowered to the boundary tier (`toBoundaryJobSchedule`), and + * the job's `retryPolicy` / `timeoutMs` are threaded to the adapter. For a body + * job the same `timeoutMs` also bounds the sandbox run — the one limit + * `JobSchema.timeoutMs` states. + * + * Re-scheduling replaces, it never accumulates (#21489), as the hook and + * action halves do: `IJobService.schedule` replaces a job of the same name, and + * every job this app scheduled on the job service before and does not schedule + * now — dropped by the new version, disabled, or no longer runnable — is + * CANCELLED ({@link retireAppJobs}). A call with no jobs at all cancels + * everything the app scheduled. The package's uninstall cancels the rest, through + * the uninstall cleanup this function registers ({@link ensureJobUninstallCleanup}). + * + * Never throws: a job that fails to schedule is logged at `error` with its own + * counter (a silent outage otherwise — the app looks healthy and the work never + * runs), and the rest still schedule. + */ +export async function scheduleAppArtifactJobs( + ctx: PluginContext, + bundle: unknown, + options: AppArtifactJobSchedulingOptions, +): Promise { + const { appId, ql } = options; + const logger: Logger = ctx.logger; + const tag = `[${options.source ?? 'AppPlugin'}]`; + const out: AppArtifactJobScheduling = { bodies: [], handlers: [], notScheduled: [], failed: [], cancelled: [] }; + + const jobs = collectBundleJobs(bundle); + let svc: IJobService | undefined; + try { svc = ctx.getService('job'); } catch { /* not installed */ } + if (svc && typeof svc.schedule !== 'function') svc = undefined; + + // Nothing to schedule: whatever this app scheduled before stops — a + // reinstall whose new version declares no jobs. + if (jobs.length === 0) { + if (svc) out.cancelled = await retireAppJobs(svc, appId, new Set(), logger, tag); + return out; + } + + // [#17396] The DEPLOYMENT gate, ahead of the job service probe. Every job + // reaching this function is PACKAGE-AUTHORED — it arrived through + // `defineStack({ jobs })` or a package bundle — which is exactly the + // boundary the switch draws. ⛔ Platform-internal scheduled work is NOT + // gated and does not pass through here: approvals escalation, the lifecycle + // Reaper, the messaging dispatch loop and membership backfill each schedule + // themselves from their own service plugin, because they are part of the + // runtime a deployment asked for rather than arbitrary load a package put on + // its clock. + // + // `info`, not `warn`: this is the default state of every deployment and the + // deployment declared it, so nothing is wrong and nothing looks + // normal-but-broken. Said once per app with the job count, rather than once + // per job — the remedy is one variable, and repeating it N times is how a + // line stops being read. + if (!resolveScheduledWorkEnabled()) { + logger.info(`${tag} declarative jobs NOT scheduled — ${SCHEDULED_WORK_DISABLED_REASON}`, { + appId, + jobCount: jobs.length, + }); + if (svc) out.cancelled = await retireAppJobs(svc, appId, new Set(), logger, tag); + return { ...out, withheld: 'scheduled-work-disabled' }; + } + if (!svc) { + logger.warn(`${tag} job service not registered — skipping declarative jobs`, { appId, jobCount: jobs.length }); + return { ...out, withheld: 'no-job-service' }; + } + const jobService: IJobService = svc; + ensureJobUninstallCleanup(ctx, jobService); + + const fnMap = collectBundleFunctions(bundle); + // The RESOLVED view a `handler` job is handed as `ctx.bundle`, not the raw + // bundle: a handler reading `ctx.bundle.objects` on a multi-package option-B + // artifact would otherwise read `undefined` with nothing thrown (ADR-0130 + // D4, #15005). Identical reference on every bundle that carries no + // `packages[]`. + const collections = resolveArtifactCollections(bundle); + const bodyRunner = jobBodyRunnerFactory(new QuickJSScriptRunner(), { ql, logger, appId }); + const metrics = resolveMetrics(ctx); + + for (const job of jobs) { + const jobName: string = job?.name; + if (!jobName) { + logger.warn(`${tag} skipping job without name`, { appId, job }); + out.notScheduled.push(String(jobName)); + continue; + } + if (job.enabled === false) { + logger.debug(`${tag} job disabled — skipping`, { appId, job: jobName }); + continue; + } + + let run: JobHandler; + let form: 'body' | 'handler'; + if (job.body) { + // The body wins over a `handler` beside it, as for hooks. When it + // cannot be bound the factory has said why, and nothing runs. + const bound = bodyRunner(job); + if (!bound) { + out.notScheduled.push(jobName); + continue; + } + run = bound; + form = 'body'; + } else { + const handler = typeof job.handler === 'string' ? fnMap[job.handler] : undefined; + if (typeof handler !== 'function') { + logger.warn( + `${tag} job handler not found in bundle.functions — skipping. A job's \`handler\` names a ` + + '`defineStack({ functions })` entry, which only a boot that loads the artifact\'s runtime module ' + + '(`os start --artifact`) carries; give the job a `body` to run it on every door.', + { appId, job: jobName, handler: job.handler }, + ); + out.notScheduled.push(jobName); + continue; + } + // #14094: the handler is given DATA REACH. A job has no graph — no + // node before it, none after — so unlike a flow `script` node it + // cannot be a pure value-returner whose I/O the surrounding graph + // performs. `ql` is the same engine handle `defineStack({ onEnable })` + // gets, and it is the only route that survives the ARTIFACT path: an + // artifact carries no `onEnable` and `mergeRuntimeModule` merges only + // `functions`, so the module-scope-global escape is never bound on an + // artifact-served boot. Additive — see `JobHandlerContext`. + run = async (jobCtx: any) => { + const jobContext: JobHandlerContext = { + ...jobCtx, + jobId: jobName, + bundle: collections, + ql: ql as IObjectQLEngine, + logger, + }; + // #14256: RETURN the handler's resolved value. `JobHandler` is + // `(context) => Promise` and all three + // shipped adapters map a resolved `{ outcome: 'degraded', reason }` + // onto a `sys_job_run.status` distinct from `success` + // (#6617/#5548). A handler that resolves `undefined` — every + // handler written before #6617 — is the `success` branch. + return await handler(jobContext); + }; + form = 'handler'; + } + + try { + await jobService.schedule( + jobName, + // #4567: authoring tier → boundary tier. `job.schedule` is the + // PARSED `Schedule`, whose cron `expression` is the ADR + // expression envelope `{dialect,source}`; `IJobService.schedule` + // (and croner behind it) take a bare cron string. + toBoundaryJobSchedule(job.schedule, jobName), + run, + // #3494: thread the authored retryPolicy/timeoutMs to the adapter. + (job.retryPolicy || job.timeoutMs) + ? { retryPolicy: job.retryPolicy, timeoutMs: job.timeoutMs } + : undefined, + ); + (form === 'body' ? out.bodies : out.handlers).push(jobName); + claimJobName(jobService, appId, jobName); + } catch (err: any) { + out.failed.push(jobName); + // #4567: a job that fails to schedule is a SILENT OUTAGE — the app + // builds and boots green while the work never runs. It gets error + // level plus its own counter, and deliberately NOT the `warn` that + // "handler not found" / "job disabled" use: those describe a job + // that was never going to run, this one describes a job the author + // is owed. + logger.error( + `${tag} Background job FAILED TO SCHEDULE — it will never run`, + err as Error, + { appId, job: jobName, schedule: job.schedule }, + ); + metrics.counter(SEMCONV.jobScheduleFailuresTotal, { app: appId, job: jobName }); + } + } + + out.cancelled = await retireAppJobs(jobService, appId, new Set([...out.bodies, ...out.handlers]), logger, tag); + + const scheduled = out.bodies.length + out.handlers.length; + logger.info(`${tag} Scheduled background jobs`, { + appId, + count: scheduled, + bodies: out.bodies.length, + handlers: out.handlers.length, + failed: out.failed.length, + cancelled: out.cancelled.length, + }); + if (out.failed.length > 0) { + logger.error( + `${tag} Some background jobs are declared but NOT scheduled`, + undefined, + { appId, scheduled, failed: out.failed.length }, + ); + } + return out; +} + +// ─── Cancelling a package's jobs: replace, and uninstall (#21489) ─────── + +/** + * Which job names each app's scheduling put on a job service, keyed by the job + * service instance — one per kernel, so two kernels in one process never share + * a record, and a record goes with its kernel. + * + * A job's `name` is its identity on the job service, and it is not namespaced + * by package: the LAST app to schedule a name owns it ({@link claimJobName}), + * so cancelling one app's jobs never stops a job another app scheduled under + * the same name afterwards. + */ +const SCHEDULED_BY_APP = new WeakMap>>(); + +function scheduledByApp(svc: IJobService): Map> { + let record = SCHEDULED_BY_APP.get(svc); + if (!record) { + record = new Map(); + SCHEDULED_BY_APP.set(svc, record); + } + return record; +} + +/** `appId` now owns `jobName` on `svc`; no other app's record still claims it. */ +function claimJobName(svc: IJobService, appId: string, jobName: string): void { + const record = scheduledByApp(svc); + for (const [owner, names] of record) { + if (owner !== appId) names.delete(jobName); + } + const mine = record.get(appId) ?? new Set(); + mine.add(jobName); + record.set(appId, mine); +} + +/** + * Cancel every job `appId` scheduled on `svc` that is not in `keep`, through + * `IJobService.cancel` — the verb every adapter implements (the cron adapter + * stops its timer, the DB adapter also marks the `sys_job` row inactive). + * Returns the names cancelled. + * + * A cancel that throws leaves its job RUNNING for a package that no longer + * declares it — code that should have stopped keeps executing while the + * install answered success — so it is logged at `error`, and the name stays on + * the record for the next replace or the uninstall to retry. + */ +async function retireAppJobs( + svc: IJobService, + appId: string, + keep: ReadonlySet, + logger: Logger, + tag: string, +): Promise { + const record = scheduledByApp(svc); + const previous = record.get(appId); + const cancelled: string[] = []; + const remaining = new Set(keep); + for (const name of previous ?? []) { + if (keep.has(name)) continue; + try { + await svc.cancel(name); + cancelled.push(name); + } catch (err: any) { + remaining.add(name); + logger.error( + `${tag} a job its package no longer schedules could NOT be cancelled — it keeps running until the runtime restarts`, + err as Error, + { appId, job: name }, + ); + } + } + if (remaining.size > 0) record.set(appId, remaining); + else record.delete(appId); + if (cancelled.length > 0) { + logger.info(`${tag} Cancelled background jobs the package no longer schedules`, { appId, jobs: cancelled }); + } + return cancelled; +} + +/** The uninstall cleanup's name on the protocol's registry (#21490). */ +export const PACKAGE_JOBS_UNINSTALL_CLEANUP = 'runtime.package-jobs'; + +/** The protocols this process registered the job cleanup on — once each. */ +const JOB_CLEANUP_REGISTERED = new WeakSet(); + +/** + * Register, once per kernel, the uninstall cleanup that cancels a package's + * scheduled jobs (#21489): `runtime.package-jobs`, on the protocol's + * uninstall-cleanup registry (`registerUninstallCleanup`, #21490). + * + * Through the registry rather than a door's own uninstall step, because the + * registry is the ONE place an uninstall's data-plane revocation runs from: + * the protocol's `deletePackage` and install-local's `DELETE` both run every + * registered cleanup with the package id, so both now stop the package's jobs + * with no per-door copy. Before this, an install-local `DELETE` answered 200 + * and the uninstalled package's job body kept executing, with a system-scoped + * `ctx.api`, until the runtime restarted. + * + * Registered from here, the moment a package's jobs are first scheduled on a + * kernel, because scheduling is what creates something to cancel; the record + * the cleanup reads is the one {@link claimJobName} keeps. The package id the + * cleanup receives is the app id the jobs were scheduled under: install-local + * passes the manifest id, which is what it schedules under. A protocol without + * the registry registers nothing, and says nothing here: the uninstall door + * that runs cleanups is the one that reports a missing runner. + * + * The cleanup never throws: a job it could not cancel is an outcome + * (`success: false`, the names in `error`), which both doors report on their + * uninstall response. + */ +function ensureJobUninstallCleanup(ctx: PluginContext, svc: IJobService): void { + let protocol: { registerUninstallCleanup?: (name: string, cleanup: (args: { packageId: string }) => Promise<{ success: boolean; removed: number; error?: string }>) => void } | undefined; + try { protocol = ctx.getService('protocol'); } catch { return; } + if (!protocol || typeof protocol.registerUninstallCleanup !== 'function') return; + if (JOB_CLEANUP_REGISTERED.has(protocol)) return; + JOB_CLEANUP_REGISTERED.add(protocol); + const logger = ctx.logger; + protocol.registerUninstallCleanup(PACKAGE_JOBS_UNINSTALL_CLEANUP, async ({ packageId }) => { + const before = [...(scheduledByApp(svc).get(packageId) ?? [])]; + if (before.length === 0) return { success: true, removed: 0 }; + const cancelled = await retireAppJobs(svc, packageId, new Set(), logger, '[uninstall]'); + const left = before.filter((name) => !cancelled.includes(name)); + return left.length === 0 + ? { success: true, removed: cancelled.length } + : { + success: false, + removed: cancelled.length, + error: `${left.length} job(s) of the uninstalled package could not be cancelled and keep running until the runtime restarts: ${left.join(', ')}`, + }; + }); +} diff --git a/packages/runtime/src/app-plugin.ts b/packages/runtime/src/app-plugin.ts index 3e5ecf8c85..4caef82e7c 100644 --- a/packages/runtime/src/app-plugin.ts +++ b/packages/runtime/src/app-plugin.ts @@ -13,28 +13,21 @@ import { type ArtifactGrantBinding, } from './security/artifact-granted-permissions.js'; import { applyArtifactForwardConversions, assertProtocolCompat } from '@objectstack/metadata-core'; -import { - resolveTenancyPosture, - resolveScheduledWorkEnabled, - SCHEDULED_WORK_DISABLED_REASON, -} from '@objectstack/types'; +import { resolveTenancyPosture } from '@objectstack/types'; import { postureEnforcesWall, type TenancyPosture } from '@objectstack/spec/security'; import { SeedLoaderService } from './seed-loader.js'; import { recordSeedOutcome } from './seed-summary.js'; import { mergeSeedDatasets, readSeedDatasets, registerSeedReplayerOnce } from './seed-datasets.js'; import { declareSeedSource } from './seed-settlement.js'; import { loadDisabledPackageIds } from './package-state-store.js'; -import type { IJobService, IMetadataService, IObjectQLEngine, II18nService } from '@objectstack/spec/contracts'; +import type { IMetadataService, IObjectQLEngine, II18nService } from '@objectstack/spec/contracts'; import { normalizeFlowFunctionEntry, type NormalizedFlowFunction } from '@objectstack/spec/automation'; import { readServiceSelfInfo } from '@objectstack/spec/api'; import { SEED_WRITE_EXECUTION_CONTEXT } from '@objectstack/spec/kernel'; import { QuickJSScriptRunner } from './sandbox/quickjs-runner.js'; import { hookBodyRunnerFactory, actionBodyRunnerFactory } from './sandbox/body-runner.js'; -import { bindAppArtifactHandlers } from './app-artifact-handlers.js'; -import { toBoundaryJobSchedule } from './job-schedule.js'; -import type { JobHandlerContext } from './job-handler-context.js'; -import { countServerTiming, SEMCONV } from '@objectstack/observability'; -import { resolveMetrics } from './observability/observability-service-plugin.js'; +import { bindAppArtifactHandlers, scheduleAppArtifactJobs } from './app-artifact-handlers.js'; +import { countServerTiming } from '@objectstack/observability'; /** * The write options every seed insert must use — the shared @@ -1117,157 +1110,18 @@ export class AppPlugin implements Plugin { // ── Auto-register declarative Background Jobs ──────────────────── // Jobs declared via `defineStack({ jobs })` are scheduled against the // running `IJobService` on `kernel:ready` (so the service plugin and - // ObjectQL engine have had a chance to register). Handler strings are - // resolved through `collectBundleFunctions(bundle)` — the same - // registry used by hooks/actions, keeping the surface uniform. + // ObjectQL engine have had a chance to register). + // + // [#21489] Through `scheduleAppArtifactJobs` — the binder's job half, + // which the install-local plugin also calls for an installed package on + // install and on rehydrate. This block used to BE that loop, resolving + // `fnMap[job.handler]` only: a job's sandboxed `body` was skipped at + // warn, and ignored when a `handler` stood beside it. The binder runs + // the body, and the body wins. See `./app-artifact-handlers.ts`. try { - const jobs: any[] = Array.isArray(this.collections.jobs) - ? this.collections.jobs - : Array.isArray((this.bundle.manifest || {}).jobs) - ? (this.bundle.manifest as any).jobs - : []; - if (jobs.length > 0) { + if (collectBundleJobs(this.bundle).length > 0) { ctx.hook('kernel:ready', async () => { - // [#17396] The DEPLOYMENT gate, ahead of the job service - // probe. Every `defineJob` reaching this loop is - // PACKAGE-AUTHORED — it arrived through `defineStack({ jobs })` - // or a package bundle — which is exactly the boundary the - // switch draws. ⛔ Platform-internal scheduled work is NOT - // gated and does not pass through here: approvals - // escalation, the lifecycle Reaper, the messaging dispatch - // loop and membership backfill each schedule themselves - // from their own service plugin, because they are part of - // the runtime a deployment asked for rather than arbitrary - // load a package put on its clock. - // - // `info`, not `warn`: this is the default state of every - // deployment and the deployment declared it, so nothing is - // wrong and nothing looks normal-but-broken. Said once per - // app with the job count, rather than once per job — the - // remedy is one variable, and repeating it N times is how a - // line stops being read. - if (!resolveScheduledWorkEnabled()) { - ctx.logger.info( - `[AppPlugin] declarative jobs NOT scheduled — ${SCHEDULED_WORK_DISABLED_REASON}`, - { appId, jobCount: jobs.length }, - ); - return; - } - let svc: IJobService | undefined; - try { svc = ctx.getService('job'); } catch { /* not installed */ } - if (!svc || typeof svc.schedule !== 'function') { - ctx.logger.warn('[AppPlugin] job service not registered — skipping declarative jobs', { - appId, jobCount: jobs.length, - }); - return; - } - const fnMap = collectBundleFunctions(this.bundle); - const metrics = resolveMetrics(ctx); - let ok = 0; - let failed = 0; - for (const job of jobs) { - const jobName: string = job?.name; - if (!jobName) { - ctx.logger.warn('[AppPlugin] skipping job without name', { appId, job }); - continue; - } - if (job.enabled === false) { - ctx.logger.debug('[AppPlugin] job disabled — skipping', { appId, job: jobName }); - continue; - } - const handler = fnMap[job.handler]; - if (typeof handler !== 'function') { - ctx.logger.warn('[AppPlugin] job handler not found in bundle.functions — skipping', { - appId, job: jobName, handler: job.handler, - }); - continue; - } - try { - await svc.schedule( - jobName, - // #4567: authoring tier → boundary tier. `job.schedule` - // is the PARSED `Schedule`, whose cron `expression` is - // the ADR expression envelope `{dialect,source}`; - // `IJobService.schedule` (and croner behind it) take a - // bare cron string. Same seam, same place, as the - // retryPolicy/timeout threading just below. - toBoundaryJobSchedule(job.schedule, jobName), - // #14094: the handler is given DATA REACH. A job has no - // graph — no node before it, none after — so unlike a - // flow `script` node it cannot be a pure value-returner - // whose I/O the surrounding graph performs. `ql` is the - // same engine handle `defineStack({ onEnable })` gets, and - // it is the only route that survives the ARTIFACT path: - // an artifact carries no `onEnable` and `mergeRuntimeModule` - // merges only `functions`, so the module-scope-global - // escape is never bound on an artifact-served boot. - // Additive — see `JobHandlerContext` for the full argument. - async (jobCtx: any) => { - const jobContext: JobHandlerContext = { - ...jobCtx, - jobId: jobName, - // The RESOLVED view, not `this.bundle`: - // a handler reading `ctx.bundle.objects` - // on a multi-package option-B artifact - // would otherwise read `undefined` with - // nothing thrown (ADR-0130 D4, #15005). - // Identical reference on every bundle - // that carries no `packages[]`. - bundle: this.collections, - ql, - logger: ctx.logger, - }; - // #14256: RETURN the handler's resolved - // value. `JobHandler` is - // `(context) => Promise` - // and all three shipped adapters map a - // resolved `{ outcome: 'degraded', reason }` - // onto a `sys_job_run.status` distinct from - // `success` (#6617/#5548). A block-bodied - // arrow that only awaited made this wrapper - // a `Promise`, so the third outcome - // was unreachable from `defineJob`: a job - // that ran to completion while its work did - // not happen was recorded as `success` with - // `reason` dropped, and the three-outcome - // table in `content/docs/automation/jobs.mdx` - // was false on the declarative door. - // A handler that resolves `undefined` — every - // handler written before #6617 — still returns - // `undefined` here, which is the `success` - // branch exactly as before. - return await handler(jobContext); - }, - // #3494: thread the authored retryPolicy/timeoutMs to the adapter - (job.retryPolicy || job.timeoutMs) - ? { retryPolicy: job.retryPolicy, timeoutMs: job.timeoutMs } - : undefined, - ); - ok++; - } catch (err: any) { - failed++; - // #4567: a job that fails to schedule is a SILENT OUTAGE — - // the app builds and boots green while the work never runs. - // It gets error level plus its own counter, and deliberately - // NOT the `warn` that "handler not found" / "job disabled" - // use: those describe a job that was never going to run, - // this one describes a job the author is owed. - ctx.logger.error( - '[AppPlugin] Background job FAILED TO SCHEDULE — it will never run', - err as Error, - { appId, job: jobName, schedule: job.schedule }, - ); - metrics.counter(SEMCONV.jobScheduleFailuresTotal, { app: appId, job: jobName }); - } - } - ctx.logger.info('[AppPlugin] Scheduled background jobs', { appId, count: ok, failed }); - if (failed > 0) { - ctx.logger.error( - '[AppPlugin] Some background jobs are declared but NOT scheduled', - undefined, - { appId, scheduled: ok, failed }, - ); - } + await scheduleAppArtifactJobs(ctx, this.bundle, { appId, ql, source: 'AppPlugin' }); }); } } catch (err: any) { @@ -2156,6 +2010,20 @@ export function collectBundleHooks(bundle: any): any[] { return out; } +/** + * Collect declarative `Job` definitions from a bundle (#21489) — the resolved + * top-level `jobs` (ADR-0130 D4: every package body's too), else the legacy + * `manifest.jobs`. One list or the other, never a merge: the read `AppPlugin` + * always made, moved here so the binder's job half and the install-local job + * gate read the jobs the boot reads. + */ +export function collectBundleJobs(bundle: any): any[] { + const stack = resolveArtifactCollections(bundle) as any; + if (Array.isArray(stack?.jobs)) return stack.jobs; + const manifest = bundle?.manifest; + return Array.isArray(manifest?.jobs) ? manifest.jobs : []; +} + /** * Collect declarative actions from the bundle. Walks both root-level * `actions[]` and per-object `objects[*].actions[]`, attaching the parent diff --git a/packages/runtime/src/index.ts b/packages/runtime/src/index.ts index 9cdee7f7ce..8fbadd5d6b 100644 --- a/packages/runtime/src/index.ts +++ b/packages/runtime/src/index.ts @@ -62,8 +62,17 @@ export { AppPlugin, collectBundleHooks, collectBundleFunctions, collectBundleFun // [#21321] The ONE binder of an app artifact's script-action bodies and body // hooks, under the owner `app:` — called by `AppPlugin.start` and by the // install-local plugin (`@objectstack/cloud-connection`) on install and rehydrate. -export { bindAppArtifactHandlers, appArtifactHandlerOwner } from './app-artifact-handlers.js'; -export type { AppArtifactHandlerBinding, AppArtifactHandlerBindingOptions } from './app-artifact-handlers.js'; +// [#21489] …and its job half: `scheduleAppArtifactJobs` schedules a package's +// jobs (a `body` runs sandboxed on every door), and `collectJobsWithoutBody` +// names the enabled jobs no JSON door can run, which install-local refuses. +export { bindAppArtifactHandlers, appArtifactHandlerOwner, scheduleAppArtifactJobs, collectJobsWithoutBody } from './app-artifact-handlers.js'; +export type { + AppArtifactHandlerBinding, + AppArtifactHandlerBindingOptions, + AppArtifactJobScheduling, + AppArtifactJobSchedulingOptions, + JobWithoutBody, +} from './app-artifact-handlers.js'; // #14094 — what a DECLARATIVE job's handler is invoked with. A job has no graph, // so unlike a flow `script` node it is given data reach (`ql`) instead of being a // pure value-returner whose I/O the surrounding graph performs. diff --git a/packages/runtime/src/sandbox/body-runner.ts b/packages/runtime/src/sandbox/body-runner.ts index 221d501f36..eadb9e70b1 100644 --- a/packages/runtime/src/sandbox/body-runner.ts +++ b/packages/runtime/src/sandbox/body-runner.ts @@ -1,11 +1,13 @@ // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. /** - * Hook & Action Body Runner Factory + * Hook, Action & Job Body Runner Factory * * Bridges the metadata-only `Hook.body` / `Action.body` discriminated union * (defined in `@objectstack/spec/data/hook-body.zod`) into an executable - * handler registered on the ObjectQL engine. + * handler registered on the ObjectQL engine — and, since #21489, a job's + * `Job.body` (the same shape, L2 only) into the handler `IJobService.schedule` + * takes ({@link jobBodyRunnerFactory}). * * The runtime owns this bridge — `objectql` itself never imports the * sandbox engine, so it can stay light enough to embed in tooling and @@ -43,8 +45,9 @@ */ import type { Hook } from '@objectstack/spec/data'; -import { HookBodySchema } from '@objectstack/spec/data'; -import type { ScriptRunner, ScriptContext, ScriptResult } from './script-runner.js'; +import { HookBodySchema, ScriptBodySchema } from '@objectstack/spec/data'; +import type { JobHandler, JobRunOutcome } from '@objectstack/spec/contracts'; +import type { ScriptRunner, ScriptContext, ScriptResult, ScriptOrigin } from './script-runner.js'; // The record-title contract, imported rather than re-derived (#11293). The // object's `nameField` pointer, a formula title's server-side evaluation and // the "what does this reference field point at" rule all have exactly one @@ -117,7 +120,7 @@ interface FactoryOptions { */ function buildBodyLogSurface( opts: FactoryOptions, - origin: { kind: 'hook' | 'action'; name: string }, + origin: { kind: ScriptOrigin['kind']; name: string }, ): ScriptContext['log'] { const logger = opts.logger; const label = `[${origin.kind} '${origin.name}']`; @@ -474,6 +477,110 @@ export function actionBodyRunnerFactory( }; } +/** + * Job body runner factory (#21489) — the ONE point a job's `body` + * (`JobSchema.body`: the hook body shape, L2 only) becomes the `JobHandler` + * handed to `IJobService.schedule`. Its caller is the binder's job half, + * `scheduleAppArtifactJobs` (`../app-artifact-handlers.ts`), which every door + * that brings an artifact in calls — the boot and install-local alike. + * + * ## What a job body receives + * + * `ctx.api`, `ctx.log` and `ctx.crypto`, each behind the capability token the + * body declares, and nothing else: that is the surface `JobSchema.body` + * declares, so nothing is added here. Not the job's name and not a manual + * trigger's `data` — the in-process `JobHandlerContext` carries both, a body + * does not until the contract declares them. + * + * `ctx.api` runs as SYSTEM ({@link buildJobSandboxContext}): a job has no + * caller, so there is no envelope to elevate, and identity-less is the posture + * #3914 measured as worse than either coherent one. What bounds a body is its + * declared `capabilities` and the stored-metadata boundary every body's api + * carries ({@link buildSandboxApi}). + * + * ## The time limit + * + * The job's own `timeoutMs` reaches the runner as `opts.timeoutMs` — the ONE + * limit of a body job (`JobSchema.timeoutMs`). `body.timeoutMs` is refused on a + * job by the spec, so a body carrying one never parsed; it is refused here as + * well, rather than run under a second limit. A job with no `timeoutMs` gets + * the runner's job default. + * + * ## What it resolves + * + * The body's return value is read as the job's `JobRunOutcome` + * (`contracts/job-service.ts`, the only reader of a job's return value), copied + * only in that declared shape: any other value is a plain success, exactly like + * an in-process handler resolving `undefined`. A throw rejects — the adapters' + * `failed`, and the retry policy's trigger. + * + * Returns `undefined` — after a `warn` naming why — when the body cannot be + * bound. The binder then schedules NOTHING for the job: a present `body` wins + * over a `handler`, so falling back to the handler here would run code the + * author replaced. + */ +export function jobBodyRunnerFactory( + runner: ScriptRunner, + opts: FactoryOptions, +): (job: { name: string; body?: unknown; timeoutMs?: number }) => JobHandler | undefined { + return (job) => { + const raw = job.body; + if (!raw) return undefined; + + const parsed = ScriptBodySchema.safeParse(raw); + if (!parsed.success) { + opts.logger?.warn?.('[BodyRunner] invalid job.body shape — the job is NOT scheduled', { + appId: opts.appId, + job: job.name, + issues: parsed.error.issues.slice(0, 3), + }); + return undefined; + } + const body = parsed.data; + if (body.timeoutMs !== undefined) { + opts.logger?.warn?.( + `[BodyRunner] job '${job.name}' carries \`body.timeoutMs\`, which a job does not accept — the job is NOT ` + + "scheduled. A job's time limit is the job's own `timeoutMs`: move the value there (milliseconds, unchanged).", + { appId: opts.appId, job: job.name }, + ); + return undefined; + } + + return async function boundJobHandler(): Promise { + const sandboxCtx = buildJobSandboxContext( + opts.ql, + buildBodyLogSurface(opts, { kind: 'job', name: job.name }), + ); + try { + opts.logger?.debug?.('[BodyRunner] job fired', { appId: opts.appId, job: job.name }); + const result = await runner.run(body, sandboxCtx, { + origin: { kind: 'job', name: job.name }, + timeoutMs: job.timeoutMs, + }); + return jobRunOutcomeOf(result.value); + } catch (err: any) { + opts.logger?.error?.('[BodyRunner] sandboxed job threw', err, { + appId: opts.appId, + job: job.name, + }); + throw err; + } + }; + }; +} + +/** + * A job body's return value, read as the declared `JobRunOutcome` and nothing + * else (#21489): `{ outcome: 'completed' | 'degraded', reason?: string }` is + * copied in that shape, every other value is `undefined` — a plain success. + */ +function jobRunOutcomeOf(value: unknown): JobRunOutcome | undefined { + if (!value || typeof value !== 'object' || Array.isArray(value)) return undefined; + const v = value as Record; + if (v.outcome !== 'completed' && v.outcome !== 'degraded') return undefined; + return typeof v.reason === 'string' ? { outcome: v.outcome, reason: v.reason } : { outcome: v.outcome }; +} + /** * Report `ctx.record` writes the action path discards (#4345). * @@ -1030,6 +1137,29 @@ function buildActionSandboxContext( }; } +/** + * The sandbox context of one job-body run (#21489): `api`, `log`, `crypto` — + * the surface `JobSchema.body` declares — and no input, caller or record. + * + * `ctx.api` is served through {@link buildSandboxApi} like every body's, under + * a fresh `{ isSystem: true }` envelope. A job has no caller to spread first + * (an action body spreads its caller's, `buildActionExecutionContext`), and the + * system envelope is what an action body with no caller gets, what a hook + * body's engine api falls back to, and what a `handler` job's in-process `ql` + * amounts to. Identity-less instead would be #3914's posture: plugin-sharing + * refuses an owner-scoped write that has neither a `userId` to own it nor + * `isSystem` to bypass. Fresh per run, never a shared constant, because an + * execution envelope is a value the engine may extend (a transaction joins it). + */ +function buildJobSandboxContext(ql: any, log: ScriptContext['log']): ScriptContext { + return { + input: undefined, + api: buildSandboxApi({ executionContext: { isSystem: true } }, ql, 'job body'), + log, + crypto: globalThis.crypto, + }; +} + /** * Convert a Proxy-wrapped record into a plain object so it round-trips through * JSON cleanly. `Object.fromEntries(Object.entries(p))` triggers the proxy's diff --git a/packages/runtime/src/sandbox/quickjs-runner.ts b/packages/runtime/src/sandbox/quickjs-runner.ts index aaefa344b3..9c35efba2a 100644 --- a/packages/runtime/src/sandbox/quickjs-runner.ts +++ b/packages/runtime/src/sandbox/quickjs-runner.ts @@ -7,7 +7,7 @@ * * Responsibilities: * - L1 ExpressionBody — evaluated as a `return ()` snippet. - * - L2 ScriptBody — wrapped in `(async (ctx) => { })(ctx)` (hooks) + * - L2 ScriptBody — wrapped in `(async (ctx) => { })(ctx)` (hooks, jobs) * or `(async (input, ctx) => { })(input, ctx)` (actions). * - Hard timeout via QuickJS interrupt handler. * - Capability gating — host-side `ctx.api`, `ctx.crypto`, `ctx.log` are only @@ -50,6 +50,15 @@ import type { const DEFAULT_HOOK_TIMEOUT_MS = 250; const DEFAULT_ACTION_TIMEOUT_MS = 5000; +// [#21489] A job body whose job declares no `timeoutMs` gets the budget an +// action body gets. It is CPU time (ADR-0102 D1): the body's `ctx.api` awaits +// are not charged, so it bounds runaway script work, never a slow query. A job +// that needs more says so in `JobSchema.timeoutMs`, the one limit of a body job +// (uncapped, unlike `ScriptBody.timeoutMs`), which reaches this runner as +// `opts.timeoutMs` and wins. No env override, unlike the two above: those lift +// a floor on a loaded host for the per-request paths, and a job's own +// `timeoutMs` is already the declared place to state how long it needs. +const DEFAULT_JOB_TIMEOUT_MS = 5000; const DEFAULT_MEMORY_MB = 32; // Wall-clock backstop (ADR-0102 D1): the CPU budget bounds VM-active time, but a // body parked forever on a host call that never settles burns no CPU — the @@ -62,6 +71,11 @@ export interface QuickJSScriptRunnerOptions { hookTimeoutMs?: number; /** Default per-invocation **CPU-time** budget for actions (ms). */ actionTimeoutMs?: number; + /** + * Default per-invocation **CPU-time** budget for job bodies (ms), used only + * when the job declares no `timeoutMs`. Default 5000. + */ + jobTimeoutMs?: number; /** * Wall-clock ceiling (ms) — the backstop for a body stuck on a never-settling * host call. Effective ceiling is `max(this, cpuBudget)`, so it can never cut @@ -85,6 +99,7 @@ export class QuickJSScriptRunner implements ScriptRunner { this.opts = { hookTimeoutMs: opts.hookTimeoutMs ?? resolveSandboxTimeoutMs('hook', DEFAULT_HOOK_TIMEOUT_MS), actionTimeoutMs: opts.actionTimeoutMs ?? resolveSandboxTimeoutMs('action', DEFAULT_ACTION_TIMEOUT_MS), + jobTimeoutMs: opts.jobTimeoutMs ?? DEFAULT_JOB_TIMEOUT_MS, wallCeilingMs: opts.wallCeilingMs ?? resolveSandboxTimeoutMs('wallCeiling', DEFAULT_WALL_CEILING_MS), memoryMb: opts.memoryMb ?? DEFAULT_MEMORY_MB, }; @@ -154,7 +169,15 @@ export class QuickJSScriptRunner implements ScriptRunner { * and pushed template authors toward denormalized rollup workarounds (#1867). */ private resolveTimeout(opts: ScriptRunOptions, bodyTimeoutMs: number | undefined): number { - const def = opts.origin.kind === 'hook' ? this.opts.hookTimeoutMs : this.opts.actionTimeoutMs; + // One default per origin kind, spelled per kind rather than as a + // hook-or-else branch: a job body falling into the action branch would have + // read the action's env override as its own (#21489). + const def = + opts.origin.kind === 'hook' + ? this.opts.hookTimeoutMs + : opts.origin.kind === 'job' + ? this.opts.jobTimeoutMs + : this.opts.actionTimeoutMs; const explicit = [opts.timeoutMs, bodyTimeoutMs].filter((n): n is number => typeof n === 'number'); return explicit.length > 0 ? Math.min(...explicit) : def; } @@ -289,7 +312,10 @@ export class QuickJSScriptRunner implements ScriptRunner { : undefined; } catch (_) { globalThis.__errorInfo = undefined; } }`; - const wrapped = args.origin.kind === 'hook' + // A job body takes the hook's `(ctx)` wrapper (#21489): it has no input to + // hand in as a first parameter, and `JobSchema.body` documents `ctx` as + // the whole surface. Only an action body is `(input, ctx)`. + const wrapped = args.origin.kind !== 'action' ? `globalThis.__result = undefined; globalThis.__error = undefined; globalThis.__errorInfo = undefined; (async (ctx) => { ${args.source} })(globalThis.__ctx).then( function(v){ globalThis.__result = JSON.stringify(v === undefined ? null : v); }, diff --git a/packages/runtime/src/sandbox/script-runner.ts b/packages/runtime/src/sandbox/script-runner.ts index 66e258cc7c..0ac8193d72 100644 --- a/packages/runtime/src/sandbox/script-runner.ts +++ b/packages/runtime/src/sandbox/script-runner.ts @@ -114,11 +114,21 @@ export type ScriptUser = ActorUser | HookContext['user']; * gating, and audit logs. */ export interface ScriptOrigin { - /** Whether the body is attached to a Hook or an Action. */ - kind: 'hook' | 'action'; - /** Object the hook/action targets, when applicable. */ + /** + * What the body is attached to: a Hook, an Action, or a scheduled Job + * (`JobSchema.body`, #21489). + * + * The kind decides three things in the engine and nothing else: the + * per-invocation CPU budget a body gets when the caller states none (each + * kind has its own default), the wrapper the source runs in (an action body + * is `(input, ctx)`; a hook or a job body is `(ctx)` — a job has no input), + * and the hook-only `ctx.input` write recorder. A job body's `ctx` is + * `api` / `log` / `crypto`: no record, no caller, no trigger payload. + */ + kind: 'hook' | 'action' | 'job'; + /** Object the hook/action targets, when applicable. A job targets none. */ object?: string; - /** Hook/Action name, used in error messages and traces. */ + /** Hook/Action/Job name, used in error messages and traces. */ name: string; } @@ -479,7 +489,13 @@ export interface ScriptResult { export interface ScriptRunOptions { origin: ScriptOrigin; - /** Hard timeout for this invocation. The smaller of body.timeoutMs and this wins. */ + /** + * Hard timeout for this invocation. The smaller of body.timeoutMs and this wins. + * + * For a job body this is the job's own `timeoutMs` (`JobSchema.timeoutMs`, + * the ONE limit of a body job — `body.timeoutMs` is refused on a job), so it + * is the only explicit value and it decides the budget alone. + */ timeoutMs?: number; /** Optional abort signal from the surrounding kernel. */ signal?: AbortSignal; diff --git a/packages/spec/liveness/job.json b/packages/spec/liveness/job.json index 0b1e68ca0e..4a44494d5f 100644 --- a/packages/spec/liveness/job.json +++ b/packages/spec/liveness/job.json @@ -1,12 +1,12 @@ { "type": "job", - "_note": "JobSchema. The file-authored path is healthy: `defineStack({ jobs })` → `AppPlugin.start`'s `kernel:ready` hook → IJobService.schedule → the service-job adapters honor every schedule shape (`CronJobAdapter.schedule` / `DbJobAdapter.schedule`) and `runWithPolicy` enforces retryPolicy/timeout (#3494 — these used to be parsed-but-ignored). `retryPolicy` here is the ENFORCED spelling ({maxRetries, backoffMs, backoffMultiplier}); do not confuse it with the datasource `retryPolicy`, which is dead and spells its delay differently. TYPE-LEVEL GAP CLOSED 2026-08-02 (#4509) by CLOSING THE DOOR, not building a bridge: `job` was registered `allowRuntimeCreate: true` while only the compiled bundle's `jobs` ever reached the scheduler, so a Studio-created job saved cleanly and never ran. Unlike the webhook (#3461) and email_template (#4509 item 1) disconnects, this one could not be bridged: `handler` names a function in the compiled bundle's function table (`collectBundleFunctions`), which a runtime writer does not have and cannot name — the missing piece is a handler-binding design, not an ingestion path. So `allowRuntimeCreate` AND `allowOrgOverride` are now both false (metadata-plugin.zod.ts, with the rationale block), leaving `*.job.ts` / `defineStack({ jobs })` as the supported doors. The kind stays registered: its file loader is genuinely consumed, so it still passes the ADR-0088 admission test. Seeded 2026-08-01. 2026-08-28 (commit 8cb96ec41): every LOCAL citation re-anchored to its consuming symbol — and this file is its own best argument for doing so. The 2026-08-02 note recorded that the SEEDED lines had already drifted ~25 lines and were restamped with fresh numbers; twenty-six days later every one of those fresh numbers had drifted again, this time by ~70-100 lines, onto `} else {`, `try {`, a bare `}` and a line about registering ACTIONS. Restamping is not a fix for line rot, it is the same claim with a newer date — which is the case this whole worklist rests on. The two objectui-cited entries (`label`, `description`) are left BYTE-FOR-BYTE UNTOUCHED: their evidence and producer are pinned at `objectui @aeb8424b`, a commit this container cannot reproduce, and foreign anchors are never collected by the scanner.", + "_note": "JobSchema. The file-authored path is healthy: `defineStack({ jobs })` → `AppPlugin.start`'s `kernel:ready` hook → IJobService.schedule → the service-job adapters honor every schedule shape (`CronJobAdapter.schedule` / `DbJobAdapter.schedule`) and `runWithPolicy` enforces retryPolicy/timeout (#3494 — these used to be parsed-but-ignored). `retryPolicy` here is the ENFORCED spelling ({maxRetries, backoffMs, backoffMultiplier}); do not confuse it with the datasource `retryPolicy`, which is dead and spells its delay differently. TYPE-LEVEL GAP CLOSED 2026-08-02 (#4509) by CLOSING THE DOOR, not building a bridge: `job` was registered `allowRuntimeCreate: true` while only the compiled bundle's `jobs` ever reached the scheduler, so a Studio-created job saved cleanly and never ran. Unlike the webhook (#3461) and email_template (#4509 item 1) disconnects, this one could not be bridged: `handler` names a function in the compiled bundle's function table (`collectBundleFunctions`), which a runtime writer does not have and cannot name — the missing piece is a handler-binding design, not an ingestion path. So `allowRuntimeCreate` AND `allowOrgOverride` are now both false (metadata-plugin.zod.ts, with the rationale block), leaving `*.job.ts` / `defineStack({ jobs })` as the supported doors. The kind stays registered: its file loader is genuinely consumed, so it still passes the ADR-0088 admission test. Seeded 2026-08-01. 2026-08-28 (commit 8cb96ec41): every LOCAL citation re-anchored to its consuming symbol — and this file is its own best argument for doing so. The 2026-08-02 note recorded that the SEEDED lines had already drifted ~25 lines and were restamped with fresh numbers; twenty-six days later every one of those fresh numbers had drifted again, this time by ~70-100 lines, onto `} else {`, `try {`, a bare `}` and a line about registering ACTIONS. Restamping is not a fix for line rot, it is the same claim with a newer date — which is the case this whole worklist rests on. The two objectui-cited entries (`label`, `description`) are left BYTE-FOR-BYTE UNTOUCHED: their evidence and producer are pinned at `objectui @aeb8424b`, a commit this container cannot reproduce, and foreign anchors are never collected by the scanner. 2026-10-03 (#21489): a job's sandboxed `body` is scheduled by the binder's job half (`scheduleAppArtifactJobs`, `packages/runtime/src/app-artifact-handlers.ts`) on every door that brings an artifact in — the boot and install-local (install and rehydrate) — and the install-local door refuses a package whose enabled job has no `body`, the one shape no JSON door can run.", "props": { "name": { "status": "live", - "verifiedAt": "2026-08-28", - "evidence": "packages/runtime/src/app-plugin.ts#start (`const jobName: string = job?.name` — a job without one is skipped with a warn before anything else is read, and the name is the scheduling key passed to `svc.schedule` plus the subject of every diagnostic in the loop)", - "note": "scheduling identity; a job without one is skipped loudly. 2026-08-28: RE-ANCHORED (commit 8cb96ec41) and REPOINTED — `app-plugin.ts:815` had rotted onto a bare `} else {` and `:833` onto a bare `try {`, ~98 and ~80 lines above the reads. HONEST RESIDUAL: the scheduling loop is inline in `AppPlugin.start`'s `kernel:ready` hook with no enclosing named helper, so `start` is the anchor — a weak one at text level, since almost any file contains the word. It is the true enclosing symbol and it is named as such rather than dressed up; the entries below that have a distinctive downstream consumer cite it beside `start` for exactly this reason. Re-closed by hand against 93ea19bca." + "verifiedAt": "2026-10-03", + "evidence": "packages/runtime/src/app-artifact-handlers.ts#scheduleAppArtifactJobs (`const jobName: string = job?.name` — a job without one is skipped with a warn before anything else is read, and the name is the scheduling key passed to `svc.schedule` plus the subject of every diagnostic in the loop)", + "note": "scheduling identity; a job without one is skipped loudly. 2026-08-28: RE-ANCHORED (commit 8cb96ec41) and REPOINTED — `app-plugin.ts:815` had rotted onto a bare `} else {` and `:833` onto a bare `try {`, ~98 and ~80 lines above the reads. HONEST RESIDUAL: the scheduling loop is inline in `AppPlugin.start`'s `kernel:ready` hook with no enclosing named helper, so `start` is the anchor — a weak one at text level, since almost any file contains the word. It is the true enclosing symbol and it is named as such rather than dressed up; the entries below that have a distinctive downstream consumer cite it beside `start` for exactly this reason. Re-closed by hand against 93ea19bca. 2026-10-03: REPOINTED (#21489) — the scheduling loop left `AppPlugin.start`'s `kernel:ready` hook for the binder's job half, `scheduleAppArtifactJobs`, which every door that brings an artifact in calls (the boot on `kernel:ready`, install-local on install and rehydrate). The read now has a named enclosing symbol, which retires the `start` residual." }, "label": { "status": "live", @@ -26,63 +26,61 @@ }, "schedule": { "status": "live", - "verifiedAt": "2026-08-28", - "evidence": "packages/runtime/src/job-schedule.ts#toBoundaryJobSchedule (#4567 authoring tier → boundary tier: the parsed `Schedule`'s cron `expression` is the ADR expression envelope `{dialect,source}` and is lowered to the bare string the adapters take, refusing by name rather than scheduling a wrong shape); packages/runtime/src/app-plugin.ts#start (`toBoundaryJobSchedule(job.schedule, jobName)` — the call site, and `job.schedule` again in the FAILED-TO-SCHEDULE error); packages/services/service-job/src/cron-job-adapter.ts#CronJobAdapter (`schedule()` branches all three variants: `schedule.expression` + the per-job timezone, `schedule.type === 'interval' && schedule.intervalMs`, `schedule.type === 'once' && schedule.at`); packages/services/service-job/src/db-job-adapter.ts#DbJobAdapter (`schedule()` routes the cron variant to the cron adapter and persists the shape onto sys_job via `upsertJobRow`)", - "note": "all three variants enforced: cron `expression` + per-job `timezone`, interval `intervalMs`, once `at`; the db adapter persists the shape onto sys_job. WALK BOUNDARY: a discriminated union — the gate classifies it as one property; the per-variant keys are covered by the adapter evidence above, not by ledger rows. 2026-08-28: RE-ANCHORED (commit 8cb96ec41) and REPOINTED — all three legs were wrong. `app-plugin.ts:834` had rotted onto `const actions = collectBundleActions(this.bundle)` — a DIFFERENT metadata kind's registration, which is the most misleading landing in this file because it still reads as bundle-wiring code; `cron-job-adapter.ts:71-88` and `db-job-adapter.ts:83` had both rotted onto docblocks. The entry also gains `toBoundaryJobSchedule`, the #4567 lowering seam that did not exist when this was written and is now the first thing that reads the key. Re-closed by hand against 93ea19bca." + "verifiedAt": "2026-10-03", + "evidence": "packages/runtime/src/job-schedule.ts#toBoundaryJobSchedule (#4567 authoring tier → boundary tier: the parsed `Schedule`'s cron `expression` is the ADR expression envelope `{dialect,source}` and is lowered to the bare string the adapters take, refusing by name rather than scheduling a wrong shape); packages/runtime/src/app-artifact-handlers.ts#scheduleAppArtifactJobs (`toBoundaryJobSchedule(job.schedule, jobName)` — the call site, and `job.schedule` again in the FAILED-TO-SCHEDULE error); packages/services/service-job/src/cron-job-adapter.ts#CronJobAdapter (`schedule()` branches all three variants: `schedule.expression` + the per-job timezone, `schedule.type === 'interval' && schedule.intervalMs`, `schedule.type === 'once' && schedule.at`); packages/services/service-job/src/db-job-adapter.ts#DbJobAdapter (`schedule()` routes the cron variant to the cron adapter and persists the shape onto sys_job via `upsertJobRow`)", + "note": "all three variants enforced: cron `expression` + per-job `timezone`, interval `intervalMs`, once `at`; the db adapter persists the shape onto sys_job. WALK BOUNDARY: a discriminated union — the gate classifies it as one property; the per-variant keys are covered by the adapter evidence above, not by ledger rows. 2026-08-28: RE-ANCHORED (commit 8cb96ec41) and REPOINTED — all three legs were wrong. `app-plugin.ts:834` had rotted onto `const actions = collectBundleActions(this.bundle)` — a DIFFERENT metadata kind's registration, which is the most misleading landing in this file because it still reads as bundle-wiring code; `cron-job-adapter.ts:71-88` and `db-job-adapter.ts:83` had both rotted onto docblocks. The entry also gains `toBoundaryJobSchedule`, the #4567 lowering seam that did not exist when this was written and is now the first thing that reads the key. Re-closed by hand against 93ea19bca. 2026-10-03: REPOINTED (#21489) — the scheduling loop left `AppPlugin.start`'s `kernel:ready` hook for the binder's job half, `scheduleAppArtifactJobs`, which every door that brings an artifact in calls (the boot on `kernel:ready`, install-local on install and rehydrate). The read now has a named enclosing symbol, which retires the `start` residual." }, "handler": { "status": "live", - "verifiedAt": "2026-08-28", - "evidence": "packages/runtime/src/app-plugin.ts#start (`const handler = fnMap[job.handler]` — a miss warns and SKIPS the job rather than scheduling a no-op); packages/runtime/src/app-plugin.ts#collectBundleFunctions (the bundle function table the string resolves against — the same registry hooks and actions use)", - "note": "resolved against the bundle's function map; a missing handler skips the job with a warning rather than scheduling a no-op. This resolution is ALSO why the type is closed to runtime creation (#4509): the function table is a bundle artifact, so a handler string authored at runtime has nothing to resolve against. 2026-08-28: RE-ANCHORED (commit 8cb96ec41) and REPOINTED — `app-plugin.ts:824-830` had rotted onto a comment about registering actions on `POST /api/v1/actions/...`, ~95 lines above the lookup. The `collectBundleFunctions` half was named in the note with a line (`:812`) that the ledger never bounded because notes are prose; it is now in `evidence` under its own symbol, which is the leg that carries the runtime-creation argument. Re-closed by hand against 93ea19bca." + "verifiedAt": "2026-10-03", + "evidence": "packages/runtime/src/app-artifact-handlers.ts#scheduleAppArtifactJobs (`fnMap[job.handler]`, read only when the job carries no `body` — a present `body` wins — and a miss warns and SKIPS the job rather than scheduling a no-op); packages/runtime/src/app-plugin.ts#collectBundleFunctions (the bundle function table the string resolves against — the same registry hooks and actions use)", + "note": "resolved against the bundle's function map; a missing handler skips the job with a warning rather than scheduling a no-op. This resolution is ALSO why the type is closed to runtime creation (#4509): the function table is a bundle artifact, so a handler string authored at runtime has nothing to resolve against. 2026-08-28: RE-ANCHORED (commit 8cb96ec41) and REPOINTED — `app-plugin.ts:824-830` had rotted onto a comment about registering actions on `POST /api/v1/actions/...`, ~95 lines above the lookup. The `collectBundleFunctions` half was named in the note with a line (`:812`) that the ledger never bounded because notes are prose; it is now in `evidence` under its own symbol, which is the leg that carries the runtime-creation argument. Re-closed by hand against 93ea19bca. 2026-10-03: REPOINTED (#21489) — the scheduling loop left `AppPlugin.start`'s `kernel:ready` hook for the binder's job half, `scheduleAppArtifactJobs`, which every door that brings an artifact in calls (the boot on `kernel:ready`, install-local on install and rehydrate). The read now has a named enclosing symbol, which retires the `start` residual." }, "body": { "children": { "language": { - "status": "planned", + "status": "live", "verifiedAt": "2026-10-03", "evidenceScope": "in-repo", - "evidence": "packages/spec/src/data/hook-body.zod.ts#ScriptBodySchema (`language: z.literal('js')`, whose error names why an L1 `\"expression\"` body is refused on this slot); packages/spec/src/system/job.zod.ts#JobSchema (`body` is `ScriptBodySchema` alone, not the `HookBodySchema` union)", - "note": "Validated, not run: the L2 discriminator the runtime binder will hand to the sandbox runner. Same carrier as `source`." + "evidence": "packages/runtime/src/sandbox/quickjs-runner.ts#QuickJSScriptRunner (`run()` dispatches on `body.language`; the job binder hands it only the L2 `js` body); packages/spec/src/data/hook-body.zod.ts#ScriptBodySchema (`language: z.literal('js')`, whose error names why an L1 `\"expression\"` body is refused on this slot); packages/spec/src/system/job.zod.ts#JobSchema (`body` is `ScriptBodySchema` alone, not the `HookBodySchema` union)", + "note": "The L2 discriminator: `jobBodyRunnerFactory` binds a job body only through `ScriptBodySchema`, so an L1 expression body is warned and the job is not scheduled (never its `handler` instead). 2026-10-03: FLIPPED to `live` (#21489) on the commit that binds it — the binder's job half (`scheduleAppArtifactJobs`) schedules a job `body` through `jobBodyRunnerFactory` on every door that brings an artifact in; `authorWarn` / `authorHint` dropped in the same edit, as this row's carrier note prescribed." }, "source": { - "status": "planned", + "status": "live", "verifiedAt": "2026-10-03", "evidenceScope": "in-repo", - "authorWarn": true, - "authorHint": "The runtime does not run a job `body` yet: a job is still scheduled through its `handler`, and a job with a `body` and no `handler` is skipped at boot with a warning. Keep the `body` (it is validated today and travels with the metadata) and keep a `handler` beside it until the runtime binder lands.", - "evidence": "packages/spec/src/data/hook-body.zod.ts#ScriptBodySchema (`source: z.string().min(1)`); packages/spec/src/system/job.zod.ts#JobSchema (the job slot that admits it)", - "note": "PLANNED, carrier #21489 (the runtime half of the E ruling, `Blocked-by:` this key's landing): the one binder, `bindAppArtifactHandlers`, schedules a job's `body` in the sandbox on every door that brings an artifact in. Measured at the commit that added the key: packages/runtime/src/app-plugin.ts#start resolves each job through `fnMap[job.handler]` only and never reads `job.body`, so a body-only job is skipped with the existing `job handler not found in bundle.functions` warn. The in-repo pieces cited in `evidence` validate the body; none of them RUNS it, which is what a `live` verdict would claim. No producer in this repo: a job body is authored as data (`os build` does not mint one from a `handler`'s function, see `JobSchema.body`'s TSDoc). `authorWarn` because a body-only job is declared-but-not-run until the binder lands. Flip every child to `live` on the commit that cites the binder reading `job.body`, and drop `authorWarn` / `authorHint` in that same edit (a warned `live` row reds this gate)." + "evidence": "packages/runtime/src/sandbox/body-runner.ts#jobBodyRunnerFactory (`runner.run(body, …)` with origin `{ kind: 'job' }` — the job's `body` becomes the handler `IJobService.schedule` takes); packages/runtime/src/sandbox/quickjs-runner.ts#QuickJSScriptRunner (`runScript` evaluates `body.source` in the `(ctx)` wrapper a job shares with hooks); packages/runtime/src/app-artifact-handlers.ts#scheduleAppArtifactJobs (`if (job.body)` — the body is bound and wins over a `handler` beside it)", + "note": "LIVE: a job's `body` runs in the QuickJS sandbox on its schedule — on the boot (`AppPlugin`, `kernel:ready`) and on install-local (install and rehydrate), through the one binder. No producer in this repo: a job body is authored as data (`os build` does not mint one from a `handler`'s function, see `JobSchema.body`'s TSDoc). 2026-10-03: FLIPPED to `live` (#21489) on the commit that binds it — the binder's job half (`scheduleAppArtifactJobs`) schedules a job `body` through `jobBodyRunnerFactory` on every door that brings an artifact in; `authorWarn` / `authorHint` dropped in the same edit, as this row's carrier note prescribed." }, "capabilities": { - "status": "planned", + "status": "live", "verifiedAt": "2026-10-03", "evidenceScope": "in-repo", - "evidence": "packages/spec/src/data/hook-body.zod.ts#ScriptBodySchema (`capabilities: z.array(HookBodyCapability).default([])`); packages/spec/src/system/job.zod.ts#JobSchema (the job slot that admits it)", - "note": "Validated, not enforced yet: the sandbox gates `ctx.api` / `ctx.log` on these tokens per invocation, and no job body is invoked until the binder lands. Same carrier as `source`." + "evidence": "packages/runtime/src/sandbox/quickjs-runner.ts#QuickJSScriptRunner (`runScript` passes `body.capabilities`, and the installed `ctx.api` / `ctx.log` / `ctx.crypto` refuse any call whose token the body did not declare); packages/spec/src/data/hook-body.zod.ts#ScriptBodySchema (`capabilities: z.array(HookBodyCapability).default([])`)", + "note": "Enforced per invocation: a job body reaches `ctx.api` (as system — a job has no caller), `ctx.log` and `ctx.crypto` only under its declared tokens. 2026-10-03: FLIPPED to `live` (#21489) on the commit that binds it — the binder's job half (`scheduleAppArtifactJobs`) schedules a job `body` through `jobBodyRunnerFactory` on every door that brings an artifact in; `authorWarn` / `authorHint` dropped in the same edit, as this row's carrier note prescribed." }, "timeoutMs": { "status": "planned", "verifiedAt": "2026-10-03", "evidenceScope": "in-repo", - "evidence": "packages/spec/src/system/job.zod.ts#JobSchema (`bannedKeys(['timeoutMs'])` on the body slot — parsed, then REFUSED with the prescription to move the value to the job's own `timeoutMs`); packages/spec/src/data/hook-body.zod.ts#ScriptBodySchema (the shape's `timeoutMs`, capped at 30000 for hooks and actions)", + "evidence": "packages/spec/src/system/job.zod.ts#JobSchema (`bannedKeys(['timeoutMs'])` on the body slot — parsed, then REFUSED with the prescription to move the value to the job's own `timeoutMs`); packages/spec/src/data/hook-body.zod.ts#ScriptBodySchema (the shape's `timeoutMs`, capped at 30000 for hooks and actions); packages/runtime/src/sandbox/body-runner.ts#jobBodyRunnerFactory (a body that never parsed and carries `timeoutMs` is refused at bind as well — warned, not scheduled — rather than run under a second limit)", "note": "PLANNED on the `api.inputMapping.transform` / `connector.authentication` precedent, deliberately not `dead` and not `live`: the key exists in the reused shape and is LOUDLY REFUSED on a job, because a job's time limit has one spelling, the job-level `timeoutMs` (whose describe states the relation). Nothing to chase for enforce-or-remove: it is enforced, by refusal." }, "memoryMb": { - "status": "planned", + "status": "live", "verifiedAt": "2026-10-03", "evidenceScope": "in-repo", - "evidence": "packages/spec/src/data/hook-body.zod.ts#ScriptBodySchema (`memoryMb: z.number().int().positive().max(256).optional()`, the shape the job's `body` slot reuses by reference)", - "note": "Validated, not enforced yet: the sandbox runner applies it per invocation (advisory under quickjs, as for hooks and actions), and no job body is invoked until the binder lands. Same carrier as `source`." + "evidence": "packages/runtime/src/sandbox/quickjs-runner.ts#QuickJSScriptRunner (`runScript`: `memoryMb: body.memoryMb ?? this.opts.memoryMb`, applied through `setMemoryLimit` — advisory under quickjs, as for hooks and actions); packages/spec/src/data/hook-body.zod.ts#ScriptBodySchema (`memoryMb: z.number().int().positive().max(256).optional()`)", + "note": "Applied per invocation, advisory under quickjs exactly as for hooks and actions. 2026-10-03: FLIPPED to `live` (#21489) on the commit that binds it — the binder's job half (`scheduleAppArtifactJobs`) schedules a job `body` through `jobBodyRunnerFactory` on every door that brings an artifact in; `authorWarn` / `authorHint` dropped in the same edit, as this row's carrier note prescribed." } }, - "note": "Drilled on the day the key landed: `body` is `ScriptBodySchema` behind a slot refinement, and its five keys do not share one verdict — `timeoutMs` is refused on a job while the other four wait for the runtime binder. `authorWarn` sits on `source`, the one key every body carries, so an authored body warns once." + "note": "Drilled on the day the key landed: `body` is `ScriptBodySchema` behind a slot refinement, and its five keys do not share one verdict — `timeoutMs` is refused on a job (at parse, and at bind for a body that never parsed) while the other four are live since #21489's binder." }, "retryPolicy": { "status": "live", - "verifiedAt": "2026-08-28", - "evidence": "packages/runtime/src/app-plugin.ts#start (`(job.retryPolicy || job.timeout) ? { retryPolicy: job.retryPolicy, timeout: job.timeout } : undefined` — threaded into `svc.schedule` only when the author set one); packages/services/service-job/src/run-with-policy.ts#runWithPolicy (`const policy = options?.retryPolicy` → maxRetries / backoffMs / backoffMultiplier / maxRetryDelayMs / jitter drive the retry loop); packages/services/service-job/src/run-with-policy.ts#RETRY_DEFAULTS (what an OMITTED member means since 17.0.0 — maxRetries 0, i.e. no retry unless asked for, #4661)", - "note": "maxRetries/backoffMs/backoffMultiplier all drive the exponential-backoff retry loop (delay = min(backoffMs * multiplier^(retry-1), maxRetryDelayMs), jittered when asked). Enforced since #3494. This is the `retryPolicy` the datasource ledger warns about confusing with its dead namesake. 2026-08-28: RE-ANCHORED (commit 8cb96ec41) and REPOINTED — `app-plugin.ts:838-841` had rotted onto `if (actions.length > 0 && typeof ql.registerAction === 'function')`, another ACTIONS line, and `run-with-policy.ts:58-65` onto the `JobAttemptRecorder` interface — a neighbouring type rather than the policy loop. The `RETRY_DEFAULTS` leg is new and is the one that decides what an author's silence means, which is the half a reader of this entry most needs. Re-closed by hand against 93ea19bca." + "verifiedAt": "2026-10-03", + "evidence": "packages/runtime/src/app-artifact-handlers.ts#scheduleAppArtifactJobs (`(job.retryPolicy || job.timeoutMs) ? { retryPolicy: job.retryPolicy, timeoutMs: job.timeoutMs } : undefined` — threaded into `svc.schedule` only when the author set one); packages/services/service-job/src/run-with-policy.ts#runWithPolicy (`const policy = options?.retryPolicy` → maxRetries / backoffMs / backoffMultiplier / maxRetryDelayMs / jitter drive the retry loop); packages/services/service-job/src/run-with-policy.ts#RETRY_DEFAULTS (what an OMITTED member means since 17.0.0 — maxRetries 0, i.e. no retry unless asked for, #4661)", + "note": "maxRetries/backoffMs/backoffMultiplier all drive the exponential-backoff retry loop (delay = min(backoffMs * multiplier^(retry-1), maxRetryDelayMs), jittered when asked). Enforced since #3494. This is the `retryPolicy` the datasource ledger warns about confusing with its dead namesake. 2026-08-28: RE-ANCHORED (commit 8cb96ec41) and REPOINTED — `app-plugin.ts:838-841` had rotted onto `if (actions.length > 0 && typeof ql.registerAction === 'function')`, another ACTIONS line, and `run-with-policy.ts:58-65` onto the `JobAttemptRecorder` interface — a neighbouring type rather than the policy loop. The `RETRY_DEFAULTS` leg is new and is the one that decides what an author's silence means, which is the half a reader of this entry most needs. Re-closed by hand against 93ea19bca. 2026-10-03: REPOINTED (#21489) — the scheduling loop left `AppPlugin.start`'s `kernel:ready` hook for the binder's job half, `scheduleAppArtifactJobs`, which every door that brings an artifact in calls (the boot on `kernel:ready`, install-local on install and rehydrate). The read now has a named enclosing symbol, which retires the `start` residual." }, "timeoutMs": { "status": "live", @@ -99,9 +97,9 @@ }, "enabled": { "status": "live", - "verifiedAt": "2026-08-28", - "evidence": "packages/runtime/src/app-plugin.ts#start (`if (job.enabled === false) { … continue; }` — the job is skipped at registration, before its handler is even resolved, so nothing is scheduled to no-op later)", - "note": "`enabled: false` skips scheduling entirely at registration — genuinely enforced, unlike the retired flow.active/tool.active. 2026-08-28: RE-ANCHORED (commit 8cb96ec41) and REPOINTED — `app-plugin.ts:820` had rotted onto a bare `}`. Same `start` residual as `name`: the read is inline in the `kernel:ready` hook and there is no narrower named symbol to anchor to. Re-closed by hand against 93ea19bca." + "verifiedAt": "2026-10-03", + "evidence": "packages/runtime/src/app-artifact-handlers.ts#scheduleAppArtifactJobs (`if (job.enabled === false) { … continue; }` — the job is skipped at registration, before its body or handler is even resolved, so nothing is scheduled to no-op later); packages/runtime/src/app-artifact-handlers.ts#collectJobsWithoutBody (the install-local door judges ENABLED jobs only, by the same rule)", + "note": "`enabled: false` skips scheduling entirely at registration — genuinely enforced, unlike the retired flow.active/tool.active. 2026-08-28: RE-ANCHORED (commit 8cb96ec41) and REPOINTED — `app-plugin.ts:820` had rotted onto a bare `}`. Same `start` residual as `name`: the read is inline in the `kernel:ready` hook and there is no narrower named symbol to anchor to. Re-closed by hand against 93ea19bca. 2026-10-03: REPOINTED (#21489) — the scheduling loop left `AppPlugin.start`'s `kernel:ready` hook for the binder's job half, `scheduleAppArtifactJobs`, which every door that brings an artifact in calls (the boot on `kernel:ready`, install-local on install and rehydrate). The read now has a named enclosing symbol, which retires the `start` residual." } } } diff --git a/packages/spec/liveness/state-counts/job.md b/packages/spec/liveness/state-counts/job.md index a4307aa2e0..80dfd4bbe4 100644 --- a/packages/spec/liveness/state-counts/job.md +++ b/packages/spec/liveness/state-counts/job.md @@ -12,4 +12,4 @@ committed anywhere: `check:liveness` sums the shards when it reads them. | Type | live | exp | elsewhere | dead | planned | classified | |---|---|---|---|---|---|---| -| `job` | 15 | 0 | 0 | 1 | 5 | 21 | +| `job` | 19 | 0 | 0 | 1 | 1 | 21 | diff --git a/packages/spec/src/system/job.zod.ts b/packages/spec/src/system/job.zod.ts index 105cfb6379..8736a9bdbb 100644 --- a/packages/spec/src/system/job.zod.ts +++ b/packages/spec/src/system/job.zod.ts @@ -267,7 +267,7 @@ export const JobSchema = lazySchema(() => strictObject({ + 'Preferred over `handler`: when both are present `body` wins. ' + 'It runs in the QuickJS sandbox with no module scope (no imports, no helpers or constants from the surrounding file): it reaches data only through `ctx.api` under its declared `capabilities` (`api.read` / `api.write` / `api.transaction`) and logs through `ctx.log` (`log`); the in-process handler context (`ql`, `logger`, `bundle`) does not exist there. ' + "Its time limit is the job's `timeoutMs` (see there): long-running work declares a `timeoutMs` that covers it, or splits into bounded runs that each finish within it. " - + 'The runtime binder that schedules job bodies has not landed yet: until it does a job runs through `handler`, so keep `handler` beside `body`.', + + "Every door that brings an artifact in schedules a job's `body` — the boot, and `os package install` on install and on every restart — while a `handler` is code that travels only in the artifact's runtime module and runs only on a boot that loads it (a config, or `os start --artifact`); `os package install` therefore refuses an enabled job with no `body`.", ), retryPolicy: RetryPolicySchema.optional().describe('Retry policy: failed runs (including timeouts) are retried with exponential backoff (delay = min(backoffMs * backoffMultiplier^(retry-1), maxRetryDelayMs), optionally jittered) up to maxRetries retries after the initial attempt. Omit the block for a single attempt; declaring it without `maxRetries` also means no retry since 17.0.0 — state a count to opt in.'), // Renamed from `timeout` (#14478): the unit (milliseconds) lived only in the @@ -312,8 +312,9 @@ export type JobParsed = z.infer; * source: "const open = await ctx.api.object('task').find({ where: { status: 'open' } }); ctx.log.info('open tasks', { count: open.length });", * capabilities: ['api.read', 'log'], * }, - * // Deprecated, kept beside `body` until the runtime binder runs job bodies: - * // must be registered in defineStack({ functions: { syncMetadata: () => ... } }) + * // Deprecated and optional beside `body`, which wins when both are present. It + * // names a defineStack({ functions }) entry, which only a boot that loads the + * // artifact's runtime module (a config, or `os start --artifact`) can run. * handler: 'syncMetadata', * }); * ```