Skip to content

Commit 21da749

Browse files
committed
Merge remote-tracking branch 'origin/main' into claude/issue-22385-objectui-pin-bump
2 parents 08d895d + fdfdd7e commit 21da749

12 files changed

Lines changed: 265 additions & 23 deletions

File tree

‎.changeset/22286-kernel-signal-listeners.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,4 @@ Clause-②: no
99
- **What was wrong.** With `gracefulShutdown: true` (the default) the constructor added one listener per signal and kept no handle, so `shutdown()` never removed them. Every kernel a process built left three listeners behind, and each one called `process.exit(0)` on the next signal. A process that had built and stopped five kernels and still ran a sixth exited six times on one SIGTERM, and the first exit landed before the live kernel had drained. Eleven built-and-stopped kernels also tripped Node's `MaxListenersExceededWarning`.
1010
- **What a kernel does now.** It holds its listeners from construction until it reaches `stopped`, and removes them at that point, whichever way it got there: `shutdown()` completing, the shutdown timeout, or a failed `bootstrap()`. A stopped kernel has no listener, so it cannot handle a signal. While a kernel is `stopping` its listener stays installed, so a repeated signal is absorbed and does not trigger Node's default action mid-drain.
1111
- **Who exits the process.** Only the handler whose signal started the drain, and only once. A signal that arrives while the host is already stopping the kernel through `shutdown()` is logged and left to the host. The kernel no longer exits the process in that case.
12-
- **Unchanged.** A running kernel still drains and exits with code 0 on SIGINT, SIGTERM or SIGQUIT (code 1 if `shutdown()` throws). `gracefulShutdown: false` still installs nothing. A genuine shutdown timeout still calls `process.exit(1)`.
12+
- **Unchanged.** A running kernel still drains and exits with code 0 on SIGINT, SIGTERM or SIGQUIT (code 1 if `shutdown()` throws). `gracefulShutdown: false` still installs nothing. With `gracefulShutdown: true`, a genuine shutdown timeout still calls `process.exit(1)`; a `gracefulShutdown: false` kernel no longer exits (#22335).
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
'@objectstack/core': minor
3+
---
4+
5+
fix(core)!: a `gracefulShutdown: false` kernel no longer exits the process when its teardown times out
6+
7+
Clause-②: no
8+
9+
<!-- adr-0087: not-required (no-migration-prescription) No metadata moves: no spec key, authorable spelling, export, option, type or stored shape is added, removed, renamed or re-shaped, and no stored row is read, rewritten or converted, so there is nothing for `objectstack migrate meta` to rewrite. What changes is a runtime behaviour of `ObjectKernel.shutdown()` under an existing option: on a teardown timeout a `gracefulShutdown: false` kernel returns to its host instead of calling `process.exit(1)`. The other categories are closed on facts: `@objectstack/core` publishes (not unpublished); no ADR-0087 id is named or touched (not registered or already-registered); and no TypeScript declaration moves, only a JSDoc comment (not runtime-interface-only or type-surface-only). -->
10+
11+
**BREAKING** for a host that builds `ObjectKernel` with `gracefulShutdown: false` and counted on the kernel to end the process when a teardown hung: a behaviour change, shipped as `minor` under the launch-window convention for breaking changes.
12+
13+
- **What changes.** When `shutdown()` overruns `shutdownTimeout`, a kernel built with `gracefulShutdown: false` no longer calls `process.exit(1)`. It logs the timeout at `error`, marks itself `stopped`, and `shutdown()` resolves, so the host decides whether the process exits. The hung teardown is not cancelled: it keeps running in the background and holds whatever it has not released until it finishes or the process ends.
14+
- **Why.** `gracefulShutdown: false` already meant that the host, not the kernel, handles the process signals, and the kernel lifecycle docs say such a kernel stays out of the process-management business. The timeout branch did not read the option. So a host that runs several `false` kernels in one process, one per environment, lost all of them when one environment's teardown hung.
15+
- **Unchanged.** With `gracefulShutdown: true` (the default; the kernel installs its own SIGINT, SIGTERM and SIGQUIT listeners) a genuine timeout still logs `Shutdown timed out — forcing exit` and calls `process.exit(1)`. A teardown that throws instead of hanging still never exits the process, under either setting. `shutdown()` still never rejects, and no option, key or export is added. Every kernel `@objectstack/cli` builds takes the default, so its commands behave as before.
16+
- **Who must act.** A host that sets `gracefulShutdown: false` (a server framework with its own signal handlers, or a host running one kernel per environment in one process) and relied on that exit to end a process whose teardown hung. Such a host already handles the signals itself; it now also ends the process itself. Call `process.exit()` in that host once `await kernel.shutdown()` returns, instead of waiting for the event loop to drain: `shutdown()` returns within `shutdownTimeout` whether or not the teardown finished, but a hung teardown can keep the loop alive after it. The timeout is reported in the kernel's `error` log line.
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
'@objectstack/service-automation': patch
3+
'@objectstack/objectql': patch
4+
---
5+
6+
fix(service-automation, objectql): the `runAs: 'user'` refusals and the run-setup warning say what a principal-less operation meets on both kernels
7+
8+
Clause-②: no
9+
10+
Three author-facing texts told a flow or hook author that a `runAs: 'user'` data operation with no trigger user "would execute UNSCOPED (elevated, RLS-bypassing)" if it were not refused. That holds only on a kernel with no security plugin. Since ADR-0096 D5, a kernel with `@objectstack/plugin-security` refuses an operation that carries no principal (`403 PERMISSION_DENIED`). Each text now says both: without a user the operation would carry no principal, refused by the security plugin where one is composed, unscoped where none is.
11+
12+
- `@objectstack/service-automation`: the `UnscopedRunDataAccessError` message, and the `[runAs]` warning logged at run setup for a flow whose trigger resolved no user.
13+
- `@objectstack/objectql`: the `HookUnscopedDataAccessError` message.
14+
15+
Only the wording changes. The refusals, their order, their codes (`AUTOMATION_UNSCOPED_RUN_DATA_ACCESS`, `HOOK_UNSCOPED_DATA_ACCESS` with its `403` status) and their remedy are unchanged: declare `runAs: 'system'` to make the elevation explicit and intended, or arrange for the trigger to supply a user.
Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* A kernel whose teardown times out exits the process only if it OWNS the
5+
* process (#22335, ruling B): `gracefulShutdown` true — the default, the kernel
6+
* installed the signal listeners itself — keeps the hard `exit(1)` (pinned in
7+
* kernel.test.ts, the #5274 timeout pin). `gracefulShutdown` false means the
8+
* host owns the process: the kernel logs the timeout, marks itself `stopped`,
9+
* and `shutdown()` returns to the host, which decides whether the process ends.
10+
*
11+
* Before this, the timeout branch called `process.exit(1)` without reading the
12+
* option, so a host running several `false` kernels in one process (one per
13+
* environment) lost every one of them when one teardown hung.
14+
*
15+
* `process.exit` is stubbed in every test here: the behaviour under test is
16+
* whether the kernel ends the process, and a real `exit(1)` would end the
17+
* vitest worker with it.
18+
*/
19+
20+
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
21+
import { ObjectKernel } from './kernel';
22+
import type { Plugin } from './types';
23+
24+
type ExitSpy = { mock: { calls: unknown[][] }; mockRestore: () => void };
25+
type LogMethod = (message: string, ...rest: unknown[]) => void;
26+
27+
const SHUTDOWN_TIMEOUT_MS = 20;
28+
29+
describe('ObjectKernel shutdown timeout under gracefulShutdown: false', () => {
30+
let exitSpy: ExitSpy;
31+
32+
beforeEach(() => {
33+
exitSpy = vi.spyOn(process, 'exit').mockImplementation((() => undefined) as never) as unknown as ExitSpy;
34+
});
35+
36+
afterEach(() => {
37+
exitSpy.mockRestore();
38+
});
39+
40+
const hostOwnedKernel = () =>
41+
new ObjectKernel({
42+
logger: { level: 'silent' },
43+
gracefulShutdown: false,
44+
skipSystemValidation: true,
45+
shutdownTimeout: SHUTDOWN_TIMEOUT_MS,
46+
});
47+
48+
const spyOnError = (kernel: ObjectKernel) =>
49+
vi.spyOn((kernel as unknown as { logger: { error: LogMethod } }).logger, 'error');
50+
51+
/** A plugin whose `kernel:shutdown` subscriber holds teardown open until released. */
52+
const hungTeardown = (reached: string[]) => {
53+
let release!: () => void;
54+
const gate = new Promise<void>((resolve) => { release = resolve; });
55+
const plugin: Plugin = {
56+
name: 'hung-teardown',
57+
version: '1.0.0',
58+
init: async (ctx) => {
59+
ctx.hook('kernel:shutdown', async () => {
60+
await gate;
61+
reached.push('teardown-resumed');
62+
});
63+
},
64+
destroy: async () => { reached.push('destroy'); },
65+
};
66+
return { plugin, release: () => release() };
67+
};
68+
69+
it('logs the timeout at error, ends stopped, resolves and does NOT exit the process', async () => {
70+
const reached: string[] = [];
71+
const { plugin, release } = hungTeardown(reached);
72+
const kernel = hostOwnedKernel();
73+
const errorSpy = spyOnError(kernel);
74+
await kernel.use(plugin);
75+
await kernel.bootstrap();
76+
77+
try {
78+
await expect(kernel.shutdown()).resolves.toBeUndefined();
79+
80+
expect(exitSpy.mock.calls).toEqual([]);
81+
expect(kernel.getState()).toBe('stopped');
82+
const timeoutLines = errorSpy.mock.calls.filter((c) => String(c[0]).startsWith('Shutdown timed out'));
83+
expect(timeoutLines).toHaveLength(1);
84+
expect(timeoutLines[0][1]).toBeInstanceOf(Error);
85+
// The `true` line says an exit is being forced; under `false` none is.
86+
expect(errorSpy.mock.calls.map((c) => String(c[0]))).not.toContain('Shutdown timed out — forcing exit');
87+
// The teardown was abandoned, not finished.
88+
expect(reached).toEqual([]);
89+
} finally {
90+
release();
91+
}
92+
93+
// The hung teardown keeps running in the background. When it finishes,
94+
// the kernel does not act on it: still `stopped`, still no exit, and a
95+
// second `shutdown()` is the already-stopped no-op.
96+
await vi.waitFor(() => expect(reached).toEqual(['teardown-resumed', 'destroy']));
97+
expect(kernel.getState()).toBe('stopped');
98+
await expect(kernel.shutdown()).resolves.toBeUndefined();
99+
expect(exitSpy.mock.calls).toEqual([]);
100+
errorSpy.mockRestore();
101+
});
102+
103+
it('two host-owned kernels in one process: stopping the hung one leaves the other running and the process alive', async () => {
104+
const reached: string[] = [];
105+
const { plugin, release } = hungTeardown(reached);
106+
const hung = hostOwnedKernel();
107+
const sibling = hostOwnedKernel();
108+
await hung.use(plugin);
109+
await hung.bootstrap();
110+
await sibling.bootstrap();
111+
112+
try {
113+
await hung.shutdown();
114+
115+
expect(hung.getState()).toBe('stopped');
116+
expect(sibling.getState()).toBe('running');
117+
expect(exitSpy.mock.calls).toEqual([]);
118+
119+
// The sibling is untouched: the host can still stop it normally.
120+
await sibling.shutdown();
121+
expect(sibling.getState()).toBe('stopped');
122+
expect(exitSpy.mock.calls).toEqual([]);
123+
} finally {
124+
release();
125+
}
126+
});
127+
128+
// Control: a teardown that FAILS rather than hangs never exited the process
129+
// under `false`, before this change or after it. Both shapes: a throwing
130+
// `destroy()` (isolated inside the teardown) and an error escaping the
131+
// teardown altogether (the non-timeout branch of `shutdown()`'s catch).
132+
it.each([
133+
['a throwing destroy()', (kernel: ObjectKernel) =>
134+
kernel.use({
135+
name: 'throwing-destroy',
136+
version: '1.0.0',
137+
init: async () => {},
138+
destroy: async () => { throw new Error('destroy boom'); },
139+
})],
140+
['an error escaping the teardown', async (kernel: ObjectKernel) => {
141+
vi.spyOn(kernel as unknown as { performShutdown: () => Promise<void> }, 'performShutdown')
142+
.mockRejectedValue(new Error('teardown boom'));
143+
}],
144+
] as const)('control: %s does not exit the process and is not reported as a timeout', async (_label, arrange) => {
145+
const kernel = hostOwnedKernel();
146+
const errorSpy = spyOnError(kernel);
147+
await arrange(kernel);
148+
await kernel.bootstrap();
149+
150+
await expect(kernel.shutdown()).resolves.toBeUndefined();
151+
152+
expect(exitSpy.mock.calls).toEqual([]);
153+
expect(kernel.getState()).toBe('stopped');
154+
expect(errorSpy.mock.calls.some((c) => String(c[0]).startsWith('Shutdown timed out'))).toBe(false);
155+
errorSpy.mockRestore();
156+
});
157+
});

‎packages/core/src/kernel.test.ts‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -966,10 +966,17 @@ describe('ObjectKernel', () => {
966966
// making progress, so the process would hang holding whatever it failed
967967
// to release. This behaviour is unchanged by #5274 — pinned so the
968968
// narrowing of the catch cannot quietly take it along.
969+
//
970+
// The fixture owns the process (`gracefulShutdown: true`, the default):
971+
// only a kernel that installed the signal listeners itself exits on a
972+
// timeout (#22335). The `false` half — no exit, the host decides — is
973+
// pinned in kernel-shutdown-timeout-ownership.test.ts.
969974
it('still logs the timeout and still forces exit(1) when shutdown genuinely times out (#5274)', async () => {
975+
const signals = ['SIGINT', 'SIGTERM', 'SIGQUIT'] as const;
976+
const listenersBefore = new Map(signals.map((s) => [s, process.listeners(s)]));
970977
const slowKernel = new ObjectKernel({
971978
logger: { level: 'error' },
972-
gracefulShutdown: false,
979+
gracefulShutdown: true,
973980
skipSystemValidation: true,
974981
shutdownTimeout: 20,
975982
});
@@ -1000,6 +1007,13 @@ describe('ObjectKernel', () => {
10001007
} finally {
10011008
exitSpy.mockRestore();
10021009
errorSpy.mockRestore();
1010+
// A `true` kernel listens on the worker's own process; never
1011+
// leave its listeners behind, whatever the test did.
1012+
for (const s of signals) {
1013+
for (const l of process.listeners(s)) {
1014+
if (!listenersBefore.get(s)!.includes(l)) process.removeListener(s, l);
1015+
}
1016+
}
10031017
}
10041018
}, 5000);
10051019

‎packages/core/src/kernel.ts‎

Lines changed: 51 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -80,10 +80,29 @@ export interface ObjectKernelConfig {
8080
defaultStartupTimeout?: number;
8181

8282
/**
83-
* Whether to enable graceful shutdown. When true (the default) the kernel
84-
* listens for SIGINT, SIGTERM and SIGQUIT from construction until it stops:
85-
* a signal drains it and then exits the process. Its listeners are removed
86-
* when it reaches `stopped`, so a stopped kernel never handles a signal.
83+
* Whether this kernel owns the process. It decides two things, always
84+
* together:
85+
*
86+
* - **Who installs the signal listeners.** When true (the default) the
87+
* kernel listens for SIGINT, SIGTERM and SIGQUIT from construction until
88+
* it stops: a signal drains it and then exits the process. Its listeners
89+
* are removed when it reaches `stopped`, so a stopped kernel never
90+
* handles a signal. When false the kernel installs none, and the host
91+
* handles signals itself.
92+
* - **Who exits the process when a teardown times out.** When `shutdown()`
93+
* overruns `shutdownTimeout`, a kernel with this option true logs the
94+
* timeout and calls `process.exit(1)`. A kernel with it false does NOT
95+
* exit the process: it logs the timeout at `error`, marks itself
96+
* `stopped`, and `shutdown()` returns to the host, which decides whether
97+
* the process exits. The hung teardown is not cancelled — it keeps
98+
* running in the background and holds whatever it has not released yet,
99+
* until it finishes or the host ends the process.
100+
*
101+
* Set it false when something else owns the process: a server framework
102+
* that installs its own signal handlers, or a host that runs several
103+
* kernels in one process and must not lose all of them when one teardown
104+
* hangs. Such a host exits the process itself when it decides to.
105+
* `shutdown()` never rejects under either setting.
87106
*/
88107
gracefulShutdown?: boolean;
89108

@@ -688,13 +707,34 @@ export class ObjectKernel {
688707

689708
if (error === shutdownTimeoutError) {
690709
// GENUINE timeout: `performShutdown()` is still running and has
691-
// stopped making progress, so the process would otherwise hang
692-
// holding whatever it failed to release. Hard-exit stays — it
693-
// is the only branch it was ever right for.
694-
this.logger.error('Shutdown timed out — forcing exit', error as Error);
695-
// Flush logger then hard-exit; the process would otherwise hang
696-
await this.logger.destroy();
697-
process.exit(1);
710+
// stopped making progress. Whether that ends the process
711+
// depends on who owns it, and the test is the one the
712+
// constructor uses to install the signal listeners, so a kernel
713+
// exits on a timeout exactly when it took the signals itself
714+
// (`ObjectKernelConfig.gracefulShutdown`).
715+
if (this.config.gracefulShutdown) {
716+
// The kernel owns the process, which would otherwise hang
717+
// holding whatever the teardown failed to release.
718+
// Hard-exit stays — it is the only branch it was ever
719+
// right for.
720+
this.logger.error('Shutdown timed out — forcing exit', error as Error);
721+
// Flush logger then hard-exit; the process would otherwise hang
722+
await this.logger.destroy();
723+
process.exit(1);
724+
} else {
725+
// The host owns the process. Exiting here would end every
726+
// other kernel and workload it runs over one hung
727+
// teardown, so the kernel reports, stays `stopped` and
728+
// returns. Nothing here acts on the abandoned teardown
729+
// again: `performShutdown()` changes no state when it
730+
// finishes, and the kernel's own later log lines from it
731+
// reach the console only, the file stream having been
732+
// closed in `finally`.
733+
this.logger.error(
734+
`Shutdown timed out after ${this.config.shutdownTimeout}ms — the kernel is stopped but its teardown is still running in the background and may still hold what it has not released; gracefulShutdown is false, so the host owns the process and it is NOT being exited — the host decides whether it exits`,
735+
error as Error,
736+
);
737+
}
698738
} else {
699739
// NOT a timeout. `performShutdown()` isolates every teardown
700740
// step it owns (hook dispatch, each destroy(), each shutdown

‎packages/objectql/src/hook-run-as.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -207,7 +207,7 @@ describe('#14010 Hook.runAs — the identity a hook\'s ctx.api presents', () =>
207207
expect(thrown.hook).toBe('stamp_grade');
208208
// The remedy the author can act on, and the reason.
209209
expect(thrown.message).toContain("runAs: 'system'");
210-
expect(thrown.message).toContain('UNSCOPED');
210+
expect(thrown.message).toContain('refused by the security plugin where one is composed, unscoped where none is');
211211
// The decisive half: refusing means the operation did NOT run unscoped.
212212
expect(engine.seen).toEqual([]);
213213
});

0 commit comments

Comments
 (0)