Merge remote-tracking branch 'origin/main' into claude/issue-20822-retired-matcher-pointers

Author: claude

.changeset/20827-choice-door-select-radio-needs-options.md

@@ -0,0 +1,50 @@
+---
+"@objectstack/spec": minor
+---
+
+fix(spec)!: `FieldSchema` refuses a `select` / `radio` field with neither `options` nor `picklist` (#20827)
+
+Clause-②: yes
+
+<!-- adr-0087: not-required (no-migration-prescription) A validity narrowing over two existing keys: `options` and `picklist` are not removed, renamed or re-shaped, so there is no tombstone and nothing mechanical for `objectstack migrate meta` to rewrite. Which choices a `select` / `radio` with no option source was meant to offer is authoring intent that no conversion entry can invent. Stored rows follow the ruling's arm for rows that exist (production `sys_metadata` is not measurable from this repository, so the disposition assumes some do): new writes are refused at the parse site with the remedy, and a stored row keeps its bytes, is still served with `_diagnostics.valid: false`, is listed by `/meta/diagnostics`, and is counted invalid and named by the `field/choice-without-options` boot line until an option or a picklist is added (pinned in `@objectstack/metadata-protocol`). The in-tree authored population is zero (examples and platform objects); the fixture census is on the PR. The other categories are closed on facts: the package publishes (not unpublished); no ADR-0087 id covers the choice-source rule and this diff adds none (not registered / already-registered); and the change narrows a metadata schema's accept set, not a runtime interface or a type surface alone (not runtime-interface-only / type-surface-only). -->
+
+**BREAKING** accept-set narrowing on `FieldSchema`, shipped as `minor` under the
+repo's launch-window convention for breaking changes — the grade the `reference`
+precedent shipped with (a `lookup` / `master_detail` without `reference`, refused
+at parse as a `minor` with the **BREAKING** header).
+
+**What was accepted before.** A `select` or `radio` field with no `options` key,
+with `options: []`, and with no `picklist` parsed cleanly. It is a choice with
+nothing to choose: the form control offers nothing, and server-side value
+validation is off (the record validator checks membership only against a
+non-empty allowed list), so any value writes through the API. The author-time
+completeness gate (ADR-0078, `field/choice-without-options`, used by `os build`,
+`os validate` and `os lint`) already graded it an error, and registration warns on
+it; a runtime-API or Studio save was the one door that let it through.
+
+**What is refused now.** At parse, on the `options` path, with a `custom` issue
+that names the field type and both remedies: a `select` / `radio` whose `options`
+is absent or empty and whose `picklist` is absent. The predicate is the
+completeness gate's own, so the two cannot disagree.
+
+**The fix.** Declare `options: [{ label, value }]` with at least one entry, or
+`picklist: 'industry'` (the name of any shared list) to offer a shared list —
+never both (that pair stays refused as before). If any value is meant to be allowed, use a `text` field instead.
+
+**Unchanged.** `multiselect` and `tags` keep parsing without options (free-form
+by design), and `checkboxes` keeps parsing with a completeness warning. A
+`select` / `radio` with at least one option, or with a `picklist`, parses as
+before. The ADR-0078 author-time rule and the registration warning are
+unchanged — this door is one more gate, not a replacement. `Field.select()`
+called with an empty list emits `options: []`, which is now refused at parse.
+
+**Stored rows.** No conversion can supply the missing options, so a row saved
+before this release is not rewritten. It is still served — with
+`_diagnostics.valid: false` naming `fields.FIELD.options` — and still
+registered at boot (counted invalid); a later save of its object is refused
+until an option or a `picklist` is added. To find such rows, read
+`GET /api/v1/meta/diagnostics`, or the boot log's `field/choice-without-options`
+lines. The `os migrate meta --stored` preview does not validate bodies, so it
+does not find them: it counts such a row canonical, or — when the row also
+carries an older spelling to lower — pending, and the apply then reports that
+row failed and leaves its bytes as they were.

.changeset/21262-audit-failure-line.md

@@ -0,0 +1,16 @@
+---
+'@objectstack/plugin-audit': patch
+---
+
+The `Audit write FAILED` line names the table whose insert was refused and the row that is lost, gives a missing table the two causes the evidence cannot tell apart, and says it is printed once per audited object, refused table and error code
+
+Clause-②: no
+
+The record writer stores the `sys_audit_log` row that records who did it, then, when activities are enabled and the write has one, its `sys_activity` timeline row. When either insert was refused, the line always said the `sys_audit_log` row never landed. When the refused insert was `sys_activity`, every ledger row had in fact landed.
+
+- The line now opens `Audit write FAILED on TABLE` and names the table the writer had in flight when it threw. A refused `sys_activity` insert says the ledger row landed and only the activity row is lost. A refused `sys_audit_log` insert says the ledger row is lost, and so is the activity row due after it when the object writes one.
+- A missing table no longer gets only the telemetry-datasource split as its remedy. The table may never have been created because schema sync's DDL for it was refused at boot. The line cannot tell the two causes apart, so it names both, in order: look for `Schema sync FAILED for object 'TABLE'` in the boot log first, then the split and `OS_TELEMETRY_DB=0`. Any other cause keeps the driver-fault remedy.
+- Whether the table is missing is asked about the refused table first. An error code that means "missing" without a phrase naming a relation is now attributed to that table, not to `sys_audit_log` by list order.
+- The line is printed once per audited object, refused table and error code, and it now says so in place of "reported ONCE". The refused table joins the key, so the other table refusing with the same code on the same object gets its own line. The same missing table still prints one line per audited object that writes through it. Repeats stay at `debug`, which now also carries the `table`.
+
+Log text and log metadata only: no status, error code, route, row or control flow changes. A log filter that matches the old text (`Audit write FAILED (`, `reported ONCE`) needs the new spelling.

.changeset/21319-explain-json-column-operator-refusal.md

@@ -0,0 +1,20 @@
+---
+'@objectstack/plugin-security': patch
+---
+
+fix(plugin-security): `security/explain` answers the read's `INVALID_FILTER` / 400 for a row-level policy that aims an operator the read refuses at a field declared JSON-stored, instead of a "visible" verdict for a request enforcement refuses (#21319)
+
+Clause-②: no
+
+The read a row-level policy scopes refuses a scalar comparison, an ordering or a text operator (`@objectstack/core`'s `JSON_COLUMN_INCOMPATIBLE_OPERATORS`, and implicit equality) on a field the object declares JSON-stored: a structured-JSON type (`json`, `address`, …), or a multi-valued field (`tags`, `multiselect`, `checkboxes`, or a `select` / `radio` / `lookup` / `user` / `file` / `image` flagged `multiple: true`). The row-level write `check` refuses them too, by the same rule. `security/explain` (the `security` service's `explain()` and `POST /api/v1/security/explain`) evaluated them in JS instead. Measured with `SecurityPlugin` on two SQLite driver families, as a member resolving a permission set whose `using` is the predicate:
+
+| `using` | find | explain, before |
+|---|---|---|
+| `record.tags != 'x'` (`tags` is `tags`, multi-valued) | 400 | `visible: true`, decided by `rls` |
+| `record.meta == 'x'` (`meta` is `json`) | 400 | `visible: true`, decided by `rls` |
+| `!(record.tags in ['x'])` | 400 | `visible: true`, decided by `rls` |
+| `record.owners != 'x'` (a `select` or `lookup` flagged `multiple`) | 400 | `visible: true`, decided by `rls` |
+
+The report without a record id said `allowed: true`, and a record id no row carries was reported `visible: false`. Now explain answers every one of these with the read's refusal, `INVALID_FILTER` / 400 and no verdict, for every operation, the answer it already gives a policy comparing two fields of different classes; a by-id update or delete is itself refused 403, at the row-level gate whose pre-image re-read is the refused read. The message leads with the full diagnostic, which names the field and the operator and says how to repair the policy, then the policy that carries it; the error's `cause` carries the read's refusal, with the find's code, status and message. The rule is the one the write check applies, and it reads the object's declaration, never the record.
+
+Unchanged: `contains` and its negation (`$contains` / `$notContains`), and the presence checks (`== null`, `!= null`), answer on such a field as before; a field declared neither way keeps every operator; an object whose schema cannot be loaded is judged as before. To repair a refused policy, test membership with `contains` (for example `!record.tags.contains('x')`).

.changeset/21324-verify-json-stdout.md

@@ -0,0 +1,15 @@
+---
+'@objectstack/cli': patch
+---
+
+fix(cli): `os verify --json` writes exactly one JSON document to stdout; the booted stack's log lines move to stderr (#21324)
+
+Clause-②: no
+
+`os verify --json > report.json` used to exit 0 and leave a file no JSON parser accepts. On a two-object stack that reaches the runtime stage, 318 lines landed on stdout ahead of the report: the kernel logger's `INFO` and `WARN` records, the ObjectQL registry's `[Registry] …` lines and the HTTP server's stop line. `JSON.parse` failed at position 4.
+
+Under `--json`, stdout now carries the report and nothing else, and every other line the run writes goes to stderr. Nothing is dropped: the boot records, the warnings among them and the shutdown lines all still reach the operator, on stderr. The document is unchanged, and so is the shape of each of the three `--json` documents (the runtime report, the author-time refusal, and the could-not-run envelope).
+
+`os verify` without `--json` is unchanged: the log lines stay on stdout beside the text report.
+
+A script that read those log lines from `os verify --json`'s stdout now reads them from stderr.

.changeset/21333-objectql-boolean-comparand-door.md

@@ -0,0 +1,31 @@
+---
+"@objectstack/objectql": minor
+---
+
+fix(objectql)!: a comparand against a declared boolean field is narrowed to its boolean at the engine's filter door, and any string other than "true" / "false" / "1" / "0" is refused with `INVALID_FILTER` / 400
+
+Clause-②: yes (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) a refusal and a narrowing of filter COMPARANDS at the engine's query door, against a declared boolean or toggle field. No authorable key, spelling, export or stored shape moves: every query shape, every FilterCondition and every object definition parse as before, @objectstack/objectql exports nothing new and nothing less, and no stored row is read or rewritten. What is refused is a comparand that matched no row (and every row under $ne) on InMemoryDriver and on SqlDriver over SQLite (PostgreSQL and MySQL not measured), and which boolean a caller meant by "yes" is not something a ledger entry can decide. The other categories are closed on facts: the package publishes (not unpublished); no ADR-0087 id covers a filter comparand, and this diff adds none (not registered / already-registered); and the change is runtime behaviour, not a declaration (not runtime-interface-only / type-surface-only). -->
+
+**BREAKING**: this narrows what a filter may compare a declared `boolean` or `toggle` field with, at every filter position and through every door that reaches the engine's filter walk (`engine.find` / `findOne` / `count` / `aggregate` / `update` / `delete`, and every spelling the data API hands it). It ships as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
+
+**What was accepted before.** A string compared with a boolean field was neither refused nor read as a boolean: the engine handed it to the driver as written, and every answer was a 200. Measured on two rows (one `true`, one `false`) on InMemoryDriver and on SqlDriver over SQLite, through `engine.find`, `engine.aggregate` and the protocol's `findData` with each spelling the `POST /api/v1/data/:object/query` and `GET /api/v1/data/:object` routes hand it:
+
+- `"true"` (implicit, `$eq`, `$in`) and `"false"` (implicit), and both through `?filter=`, `?$filter=`, the filter AST and the bare query parameter (`?flag=true`), matched no row on either driver;
+- `$ne "true"` and `$nin ["true"]` returned both rows, the true row included;
+- `"yes"` matched no row, and `$ne "yes"` both rows;
+- `1`, `"1"`, `0` and `"0"` at `where` (and `"1"` / `"0"` through every spelling above) matched the right row on SQLite and no row on InMemoryDriver (`$ne 1` returned both rows there);
+- the per-aggregation `filter` and `having` (the engine's own evaluator) answered `"true"` with no row and no group, and `$ne "true"` with every one.
+
+**What is answered now.** At `where` (both spellings), the per-aggregation `filter` and `having`, on every verb that collects a filter, before any driver is asked for a row:
+
+- `true` / `false` are handed to the driver as written;
+- `1` / `0`, `"1"` / `"0"` and `"true"` / `"false"` are narrowed to `true` / `false`, so every driver receives the one boolean each names. Measured on InMemoryDriver and on SqlDriver over SQLite, `?flag=true` and `?flag=1` now return the true row; any other driver receives the same narrowed boolean by mechanism (PostgreSQL and MySQL not measured);
+- any other string, a different letter case (`"TRUE"`), surrounding whitespace, a blank and a `{placeholder}` included, is refused `INVALID_FILTER` / 400. The message names the field, its declared type, the comparand and its position, and says what is wrong with it.
+
+The accepted set is the one the record validator already admits when a boolean field is WRITTEN. The rule lives in `@objectstack/spec/data`'s `filter-boolean-comparand-declared-type.ts`, and the engine applies it in the same walk that judges number comparands.
+
+**The remedy.** Write `true` or `false`. In a querystring, where every value is a string, write `true` / `false` or `1` / `0`.
+
+**Unchanged.** A boolean comparand, `null` (the null test) and the flag operators (`$null`, `$exists`, `$empty`) answer as before, and so does every comparand against a field that is not boolean. A number other than `1` / `0` against a boolean field is still handed to the driver as written. A filter on a `formula` field is still refused one step earlier, as before.

.changeset/21333-spec-boolean-comparand-contract.md

@@ -0,0 +1,19 @@
+---
+"@objectstack/spec": minor
+---
+
+feat(spec): the boolean-comparand declared-type contract in `@objectstack/spec/data` — the comparands a declared boolean field accepts in a filter, the boolean each narrows to, and the refusal words
+
+Clause-②: yes
+
+**What it declares.** `filter-boolean-comparand-declared-type.ts`, the boolean twin of `filter-number-comparand-declared-type.ts`:
+
+- `BOOLEAN_COMPARAND_SPELLINGS`: the accepted non-boolean spellings, `1` / `0`, `"1"` / `"0"` and `"true"` / `"false"`, each with the boolean it narrows to. This is the set the record validator admits when a boolean field is written. `readBooleanComparand` reads a comparand by it, and names why a string is not one (`NON_BOOLEAN_STRING_FORMS`: `empty`, `padded`, `letter-case`, `placeholder`, `not-a-boolean`).
+- `BOOLEAN_COMPARAND_DOOR_JUDGED_TYPES` (`BOOLEAN_VALUE_TYPES` itself), and the judged positions, which are the number door's lists by identity.
+- `booleanComparandFieldVerdict` and `booleanComparandDoorVerdict`, the pure verdict: `narrows`, `door-refusal` (`INVALID_FILTER` / 400), `passes` or `deferred`.
+- `booleanComparandRefusalMessage`: the refusal words, inside the 500-character client bound.
+- `BOOLEAN_COMPARAND_READING_CASES`, `BOOLEAN_COMPARAND_DOOR_FIXTURE` and the derived `BOOLEAN_COMPARAND_DOOR_CASES`, for a door's suite to drive.
+
+**What the verdict answers `door-refusal` for.** A string other than the four accepted ones, compared with a declared boolean field, at the value positions of a filter (the implicit comparand, `$eq` / `$ne` / `$gt` / `$gte` / `$lt` / `$lte`, and each member of `$in` / `$nin` / `$between`).
+
+**What moves for consumers.** Nothing in this package refuses or narrows a filter, and every existing export is unchanged. The door that applies the verdict ships in the same release in `@objectstack/objectql`, whose changeset states what changes for a caller.

.changeset/21345-driver-fault-redaction-residue.md

@@ -0,0 +1,14 @@
+---
+'@objectstack/objectql': patch
+---
+
+fix(objectql): a raw statement's driver fault, and a lifecycle sweep's direct-driver fault, no longer carry the statement or the caller's values
+
+Clause-②: no
+
+Two paths the engine-boundary cut did not reach now take it.
+
+- **`ObjectQL.execute`.** The cut ran on a driver error's message only when the shared leak predicate recognised a statement in it, and the predicate recognises four leading verbs. A raw statement opening with any other word, such as a common-table-expression form or a dialect's own upsert or merge verb, kept the statement and the bound values on the declared fault's `cause` (its `message` and `stack`), where any logger that prints an error's cause chain wrote them out. The door now tells the cut that it sent a statement, so the cut runs whatever word the statement opens with. The predicate's list is unchanged.
+- **The lifecycle sweep.** The Archiver copies rows to the cold store and deletes them from the hot store through the drivers directly, not through an engine door. A driver fault there, such as a cold write the archive store refused, put the archived row's values into the sweep's warning line and its `report.errors` entry. The sweep now cuts the fault the same way before it reports or logs it.
+- **What stays.** The error's class, `code`, `status` and the database's own diagnostic, on the fault and on its `cause`. A raw statement opening with one of the four recognised verbs is cut exactly as before. A sweep failure that is not a driver error is reported word for word as before.
+- **What changes for a caller.** Code that read the statement or a value out of a raw statement's fault, or out of a lifecycle sweep's error entry, now gets a `[statement and bound values redacted]` marker followed by the diagnostic. Branch on the error's class and `code` instead.