From 12f70b0956d2f7b8baf267c3d81be6ed50a3cedd Mon Sep 17 00:00:00 2001 From: azim afroozeh <13484327+azimafroozeh@users.noreply.github.com> Date: Mon, 12 Oct 2026 17:53:19 +0200 Subject: [PATCH 1/4] review --- docs/rfcs/2026-10-07-polymorphic-types.md | 816 ++++++++++++++++++ docs/rfcs/2026-10-12-distributed-sequence.md | 345 ++++++++ docs/rfcs/2026-10-12-integer-node-identity.md | 591 +++++++++++++ docs/rfcs/README.md | 3 + 4 files changed, 1755 insertions(+) create mode 100644 docs/rfcs/2026-10-07-polymorphic-types.md create mode 100644 docs/rfcs/2026-10-12-distributed-sequence.md create mode 100644 docs/rfcs/2026-10-12-integer-node-identity.md diff --git a/docs/rfcs/2026-10-07-polymorphic-types.md b/docs/rfcs/2026-10-07-polymorphic-types.md new file mode 100644 index 000000000..e7c2aef4a --- /dev/null +++ b/docs/rfcs/2026-10-07-polymorphic-types.md @@ -0,0 +1,816 @@ +--- +rfc: "2026-10-07-polymorphic-types" +title: "Polymorphic types: interfaces, unions and polymorphic edges" +track: maintainer +status: draft +implementation: not-started +authors: + - ragnorc + - azimafroozeh +created: 2026-10-07 +updated: 2026-10-12 +discussion: null +supersedes: [] +superseded_by: [] +blocked_on: + - "2026-10-12-integer-node-identity" +--- + +# RFC: Polymorphic types: interfaces, unions and polymorphic edges + +> A term set in ***bold italics*** is being defined at that exact spot. + +## Summary + +Interfaces and inline unions become ***abstract node types***: types a query +can bind, traverse to and mutate through, that name a set of concrete node +types and have no table of their own. An edge endpoint may be an interface or +a union of node types. Every node keeps exactly one concrete type, which +remains its table and its identity scope. Each abstract use resolves once per +invocation against the captured catalog to its ***type set***, the id-sorted +set of concrete member types, and the compiler and planner lower it to the +per-type pieces the engine already runs, combined by operators that carry the +concrete type as a query-only column. + +No storage changes for this RFC. An edge endpoint is already a packed node +reference, the endpoint type's `StableTypeId` in the high 16 bits and the +node's `__id` in the low 48 +([Integer node identity](2026-10-12-integer-node-identity.md)), so an edge +row whose declaration allows several endpoint types stores exactly what a +concrete edge row stores. Widening an existing endpoint from one type to an +interface or union is a schema change with no physical effect. + +Relative to the integer-identity baseline, node tables, the one-table-per-type +manifest invariant and keyed-node addressing remain unchanged. Existing +queries, loads and wire shapes that use none of this RFC's features keep their +behavior; abstract bindings, inheritance and interface constraints have the +new contracts below even without a polymorphic endpoint. + +## Motivation + +OmniGraph already parses `interface` and `node T implements I`, assigns +interfaces a `StableTypeId`, and persists them in the accepted SchemaIR. +Nothing after the schema compiler reads them. An interface is a property +template copied into each implementor; the catalog field that holds +interfaces is commented "for Phase 2 polymorphic queries". Every query binding +resolves to one concrete node type, and every edge endpoint names one +concrete node type. + +[Typed edge alternation and bounded wildcard traversal](2026-09-30-typed-edge-alternation.md) +added the first multi-type machinery for edges: member sets, common-property +typing, a synthesized type column, type-before-id ordering and budgeted +traversal. It kept one source and one destination node type per traversal and +listed heterogeneous endpoint unions and edge-union mutations as out of scope. + +Real schemas show the cost of the gap. The personal graph served at +`omnigraph.ragnor.co` declares 76 node types, 426 edge types and no +interfaces. Grouping edges that share a source type and differ only by a +destination-type suffix finds 59 families covering 202 edge types: `ExternalID` +has 23 `Identifies` edges, `Claim` has 7 `ClaimAbout` edges, `Task` has 9 +`TaskFor` edges. Four large schemas (personal, ModernRelay CRM, ModernRelay +engineering, lab knowledge core) declare no interface at all. + +The rules authors most often ask for are rules over a family, written in +comments because they cannot be declared: "the holder is a Person or an +Organization, so both edges are optional and exactly one should be present"; +"the current schema cannot express a union cardinality"; "admission requires +exactly one value edge over all target kinds", above 11 `ArgumentX` edges and a +hand-maintained `value_kind` enum. Only 13 of the 23 `Identifies*` edges +declare `@card(0..1)`, so one external id can identify a Person and an +Organization at once. + +Some questions cannot be asked in one query today. "What does this external +id identify?" needs 23 queries, because alternation refuses members with +different destination types (T5). "Find anything with slug `acme`" needs 75. +Each new type that can be identified, claimed about or tasked adds another +edge type that every query and agent prompt must learn. + +An issue or local refactor cannot fix this. The change adds a schema, query, +load and wire contract, and it touches the compiler, planner, engine, +validation and graph index. + +## User and operational behavior + +### Schema + +```pg +interface Entity @description("Anything addressed by a stable slug") { + slug: String @key + createdAt: DateTime + updatedAt: DateTime +} + +// An interface may implement interfaces. Implementors inherit the closure. +interface Identifiable implements Entity {} +interface Subject implements Entity { + name: String + brief: String? + embedding: Vector(3072)? @embed("brief", model="text-embedding-3-large") +} + +node Person implements Identifiable, Subject { email: String? } +node Organization implements Identifiable, Subject { website: String? } +node ExternalID { value: String @key } + +// Interface endpoint: open. A later implementor joins automatically. +edge Identifies: ExternalID -> Identifiable @card(0..1) + +// Inline union endpoint: closed. The edge lists its members. +edge HeldBy: Holding -> Person | Organization @card(1..1) + +// Polymorphic at both ends. +edge RelatedTo: Subject -> Subject +``` + +- `interface X implements Y, Z` declares interface inheritance. A node that + implements `X` implements the transitive closure; it need not repeat `Y` or + `Z`. +- An interface may carry type-level `@description` and `@instruction` and body + `@unique` body constraints. Body `@unique` arrives in phase 3 behind + `interface-unique` and applies across all implementors + (see [Uniqueness](#uniqueness-cardinality-and-keyed-edges)). Other body + constraint forms remain refused until their enforcement is specified. + Property-level annotations on interface properties keep today's meaning: + they are applied to each implementor that inherits the property. + Redeclaring an interface property in a node and dropping one of its + constraint annotations is a lint warning, no longer a silent drop. +- An edge endpoint is a node type, an interface, or an inline union + `A | B | …` whose members are node types or interfaces. Its member set is + the union of each member's concrete closure. +- An interface may be renamed with `@rename_from("Old")`; the rename + preserves its `StableTypeId`. +- A new schema may not declare an interface and a node or edge type with the + same name. An accepted graph that already has such a collision keeps + opening. In a query the node type wins, and the interface must be renamed + before it can be bound. +- Polymorphic endpoints require a graph at the integer identity format. A + graph from before it runs the storage upgrade first. + +### Queries + +```gq +// Bind an abstract type: matches every implementor. +query by_slug($slug: String) { + match { $x: Entity { slug: $slug } } + return { $x.@type, $x.@id, $x.createdAt } +} + +// Traverse a polymorphic edge. +query identified($value: String) { + match { + $e: ExternalID { value: $value } + $e identifies $x + } + return { $x.@type as type, $x.slug } +} + +// Narrow by rebinding and test types. +query claims_about_people() { + match { + $c: Claim + $c about $p + $p: Person + } + return { $c.text, $p.email } +} + +query non_people($q: String) { + match { + $s: Person | Organization | Project + $s.name contains $q + $s is not Person + } + return { $s } + limit 20 +} +``` + +| Form | Meaning | +|---|---| +| `$x: I` | Every concrete implementor of interface `I`, through interface inheritance. | +| `$x: A \| B` | The union of the members' sets; members may be interfaces. | +| `$x.p` | Legal when `p` is declared on `I` (or inherited by it). For a union, legal when every member has `p` under the alternation rule: same scalar and list shape, nullable if any member is, enum domains combined, enum with String becomes String. Otherwise a type error that names the narrowing to use. | +| `$x.@type` | The concrete type's canonical name, a non-null String. | +| `$x is T`, `$x is not T`, `$x is A \| B` | A non-null Bool type test. `T` must overlap the binding's set. | +| `$x: T` after an earlier binding of `$x` | Intersects the two sets. An empty intersection is a type error. Rebinding to a subtype is no longer T50. | +| `$e.@src_type`, `$e.@dst_type` | The endpoint types of a bound edge, as canonical names, read from the packed references. | +| `return { $x }` | Unchanged for a concrete binding. For a multi-member set, a struct of `@id`, `@type` and the readable properties. | + +Traversal endpoints use the same sets. A traversal's destination set is the +edge's endpoint set, intersected with any declared type of the destination +variable. An alternation's destination set is the union of its members' +destination sets, so the T5 refusal of differing destinations becomes valid +alternation. Direction is still inferred: + +- When the source's set fits only one end of the edge, that end decides. +- When it fits both ends and the edge's two endpoint sets are equal (a + same-type edge such as `Knows: Person -> Person`, or + `RelatedTo: Subject -> Subject`), the traversal is outgoing. That keeps + today's rule for same-type edges, which `test_traversal_direction_out` pins. +- When it fits both ends of unequal, overlapping sets (for example + `Mentions: Named -> Note` with `Note implements Named`), the traversal is + refused unless the other endpoint's declared type fits only one reading. + +Multi-hop selections across a polymorphic edge require each hop's destination +set to be accepted as the next hop's source set. `RelatedTo: Subject -> Subject` +recurses across Person and Organization with one shortest-distance search. + +Refusals: + +- `bm25`, `search`, `fuzzy` and `match_text` on a binding whose set has more + than one member are refused. Full-text scores are computed per dataset and + are not comparable across tables. +- `nearest` is admitted when the vector property and its `@embed` model are + declared on the interface and every member records the same model and + dimension. +- An abstract binding on an explicitly historical target is refused, matching + historical wildcard traversal. + +Allocate unused type-check codes at implementation, without reusing existing +codes (T54 and T55 are already occupied in the reviewed base). Advance the GQ +language minor version from the actual implementation base: it is 2.2 in the +reviewed tree, so 2.3 is provisional. Explain advances from 4 to 5 for union +scans; if that version is occupied before implementation, allocate the next +version. Diagnostic, language, explain and serialized-plan versions are +independent allocations. + +### Mutations and loads + +```gq +query link($ext: String, $person: String) { + insert Identifies { from: $ext, to: Person($person) } +} + +query touch($slug: String) { + update Entity set { updatedAt: now() } where slug = $slug +} +``` + +```json +{"edge":"Identifies","from":"li:alice","to":"alice","to_type":"Person"} +``` + +- `insert` into an interface or union is refused. An abstract type has no + table. +- `update I set … where …` and `delete I where …` are typed against the + interface. They apply to every member and publish once, under the existing + rule that a query cannot mix writes with deletes. Assignments may name only + properties readable on `I`; the `where` may use those properties, `@id` and + `@type`. +- A polymorphic endpoint with more than one member is written with its + concrete type: `Type($id)` in GQ, `from_type` or `to_type` in a load or + graph-batch envelope. The type must be a member of the endpoint set. + Monomorphic endpoints keep bare keys, and their envelopes refuse the new + keys, so existing strict envelopes are byte-stable. +- Export writes `from_type`/`to_type` for polymorphic endpoints, as the + identity RFC specifies. The change feed's endpoint images carry optional + `from_type`/`to_type` for them. +- A mutation `where` on a polymorphic edge may test `@src_type` and + `@dst_type`. + +### Operator-visible behavior + +- Polymorphism needs no new setting. Polymorphic traversals share the + existing `traversal_work_limit` budget. +- `omnigraph schema plan` reports the new step kinds (below) and their tier. +- An abstract binding opens one dataset per member. A cold first read pays one + dataset open per member type, and arms open concurrently within the query's + resource bounds. +- Every schema migration in this RFC is a schema apply, and schema apply + refuses while any branch other than `main` exists. A graph that keeps + long-lived branches lands them before widening an endpoint. + +## Design + +### Catalog model + +The catalog gains a `TypeSet`: an id-sorted set of concrete node types with +its provenance (`Concrete`, `Interface`, `Union`). An interface's members are +the node types whose implements-closure contains it. An endpoint's member set +is computed from its declaration and recomputed whenever the accepted schema +changes. Interface properties resolve to each member's node-owned property +through `satisfies_interface_properties`, as +[RFC 0028](0028-stable-schema-identity.md) specifies. A member's satisfying +property always carries the interface property's name, because the parser +matches by name and open compares the IR with the shape compiled from source. + +No member list is stored. The group exists only as the `implements` +declarations on node types and the sets the catalog computes from them, so a +new implementor joins every interface endpoint and every interface binding +without any other member-list record changing. Schema admission still validates +any newly introduced interface-wide constraints before publication. Interfaces, +properties and table incarnations consume the accepted schema's shared identity +allocator; the identity RFC owns its numeric type-id representability checks. + +### SchemaIR and features + +| Addition | Shape | Feature | +|---|---|---| +| Interface inheritance and annotations | `InterfaceIR.implements: Vec`, `annotations` | `interface-inheritance` | +| Interface body `@unique` | `InterfaceIR.constraints` | `interface-unique` (phase 3) | +| Abstract endpoints | An edge endpoint names a node or interface `TypeRefIR`, or a non-empty id-sorted union of them | `polymorphic-endpoints` | +| Interface rename | rename hint on `InterfaceIR`, preserving `type_id` | none (an old binary misreads nothing) | + +Each addition uses `serde(default, skip_serializing_if)`, so a graph that uses +none of them keeps byte-identical IR and the same `schema_ir_hash`. Feature +names are derived from the declarations by `required_features`, and binaries +refuse unknown names. An old binary is already fail-closed on a newer +contract: open re-serializes the IR, recomputes `schema_ir_hash` and refuses a +mismatch. The feature name makes that refusal state its reason. Validation +relaxes the endpoint-kind check in `validate_shape`, `resolve` and +`validate_schema_ir` to admit interfaces and unions, and it never derives an +identity from a name. A body `@unique` declaration derives `interface-unique` +even when the interface has no members. Binaries before phase 3 refuse the +declaration and do not advertise the feature; recognizing inheritance fields +does not imply uniqueness enforcement. Property-level constraints retain their +per-implementor meaning. + +### Physical layout + +Nothing changes. Node tables stay one Lance dataset per concrete type at the +identity-derived path. Edge tables already store each endpoint as one packed +`UInt64` whose high 16 bits are the endpoint's `StableTypeId` +([Integer node identity](2026-10-12-integer-node-identity.md)), so a +polymorphic edge table has the same columns, the same indexes and the same +row shape as a concrete one. The scalar index on each endpoint column serves +`=`, `IN` and the range predicate that selects one endpoint type, +`packed BETWEEN type << 48 AND ((type << 48) | (2^48 - 1))`. + +The identity upgrade publishes new table versions for the current snapshot. +Retained pre-upgrade versions still carry string endpoints; their decoding, +exports, change-feed images and branch/merge admission follow the identity +RFC's [historical-format contract](2026-10-12-integer-node-identity.md#retained-history). +This RFC adds no historical rewrite. Polymorphic endpoints require the +upgraded current format, and historical abstract bindings remain refused. + +### Typing and IR + +- `BoundVariable::Node` carries a `TypeSet` in place of one name, as + `BoundVariable::Edge` already carries `type_names`. `ResolvedType::Node` and + `ResolvedTraversal` endpoints carry sets. +- `resolve_member` replaces string equality with assignability: a member is + outbound when the source set is a subset of the edge's source set, subject + to the direction and ambiguity rule under [Queries](#queries). +- The multi-hop check compares sets, not binding names: a path continues when + every type a hop can end on may start the next hop. `$a: Person` over + `RelatedTo: Subject -> Subject` recurses although `Person` differs from + `Subject`. Typecheck, `ExpandStep` validation and plan admission repeat the + name comparison today; all three use one catalog predicate instead. +- `IROp::NodeScan` carries a `NodeSelection` with provenance, mirroring + `EdgeSelection`: `Named`, `Interface { name, members }`, `Union(members)`. + A polymorphic edge is one `EdgeMember` with endpoint sets; it does not + expand into one member per concrete endpoint pair. The prototype ran + forward, reverse, two-hop and negation traversals with one scan per edge + table and rows routed by their endpoint types, and no per-pair members were + needed. +- `$x.@type` lowers to a String literal for a singleton set and to a + query-only `~node_type` column otherwise, as `$e.@type` does with + `~edge_type`. A type test remains a typed, non-null Boolean expression. + Within each arm, its concrete type can replace the test with a Boolean + constant and simplify the enclosing expression. Projecting a type test + never removes rows; negation and disjunction preserve Boolean semantics. +- Lowering sites that assume a concrete type name today are unreachable, + because typecheck rejects interface names first. They change in the patch + that admits interfaces: the binding-type `.expect` in `lower.rs`, mutation + kind dispatch, and `is_scalar_string`. +- Identity is the packed node reference everywhere: sort tie-break, + deduplication, cycle closing, `AntiJoin` keys and `RankFuse` identity, for + concrete and abstract bindings alike. There is no bare-id rule for a + singleton set, because the identity RFC left no bare id to compare. The + prototype confirmed what a bare-id rule costs: lowering compared + `temp.id = dst.id`, and `not { $e identifies $x }` over an interface `$x` + silently dropped the Organization row that shares Person "alice"'s key + (finding P2). +- Narrowing has one source of truth. Today typecheck binds each binding into + two contexts and resolves traversals against the declared type while + lowering scans the checked type. Narrowing is one intersection at bind + time, and traversal resolution reads it (finding P3). +- The frozen reference engine refuses the new IR, as it refuses selections. + +### Planning and execution + +- **Union scan.** An abstract binding becomes one new logical and physical + node over N pinned table specs, one per member. It lowers through `Lower` to + one OmniGraph operator, so a plan keeps one operator per node. Each arm + carries its own pushed filter and projection, resolved through the member's + satisfying property. Each arm is cast to the binding's joined types and + gains a Utf8 `~node_type` column; the plan mirrors carry no Arrow + `Dictionary` or `Union` type. The operator streams arms and opens their + datasets concurrently. DataFusion's `UnionExec` is not used unchanged: its + output partitioning is the sum of its inputs', the engine executes only + partition 0 of the root, and it never compares input types. +- **Pruning.** A rebinding intersection removes excluded arms. A filtering + expression containing type tests or `@type` comparisons removes an arm only + when simplifying the complete filter proves that the arm cannot contribute + a row. Pruned members are never opened. +- **Cost.** Estimates sum across members. Key equality bounds each member to + one row. `count($x)` over an unfiltered abstract binding sums one + `MetadataCount` per member. +- **Expansion.** A polymorphic expansion is a selection, so it declares + `ExpandPolicy::Budgeted` and `IndexedScan` and shares the traversal-work + counter. One `src IN (frontier)` scan of an edge table serves every + destination type; rows are routed by the type half of their packed + destination. Admission charges each edge table's physical rows once per + probe window. `ExpandExec` emits `{dst}.~node_type` beside `{dst}.__id`. The + interner, frontier and visited set are keyed by the packed reference, which + makes mixed-type recursion correct where colliding keys would otherwise + merge distinct nodes. +- **The engine refuses an untyped plan.** `ExpandStep` validation refuses any + expansion that touches a polymorphic edge or an abstract endpoint unless it + runs on the typed route. In the prototype, an untyped named expansion on + the indexed route returned four rows for a two-row answer and dropped a node + from a negation, with no error (finding P8). The typechecker's routing is an + optimisation; this check is the safety mechanism. +- **A named polymorphic edge has its own policy.** It declares the budgeted + policy directly instead of posing as a one-member alternation, so a refusal + names the edge the user wrote: "edge Identifies has an interface endpoint + and cannot use traversal = csr" (finding P9). +- **Hydration.** `id_lookup` groups each slice's references by type and probes + each member table. `hash_join` builds per member type, or on the packed + key. +- **Search.** `nearest` runs per arm with `k` and merges by `_distance`. The + merge preserves each arm's accuracy, because the global top k is a subset + of the union of per-arm top-k results. The overfetch ladder runs per arm. + OmniGraph's vector indexes use L2 on every table, so distances are + comparable. +- **Graph index.** Until phase 3, polymorphic expansions never use CSR, and + the graph index build excludes polymorphic edges. `GraphIndex::build` gives + each declared endpoint name one dense id space, so a polymorphic edge would + need one space per member type and a routing step the build does not have + (finding P4). Phase 3 moves `GraphIndex` to one global ordinal space: each + node type's dense dictionary sits at a fixed offset, a node's type is + recovered from its range, and a polymorphic edge has one CSR, built by + interning each packed endpoint in the dictionary its type half names. + Admission moves to the cache owner before decoding, as the alternation RFC + requires for CSR. The persisted artifact becomes format v4. + +### Writes and validation + +- **Referential integrity** groups each delta's endpoint references by type + half, refuses a type outside the endpoint set, and runs one batched + existence probe per member type against the merged node universe. +- **Load upserts.** A Merge load with public `id` and no preservation `__id` + can address an existing unkeyed edge and move its endpoints, including their + types. Missing rows refuse, as the identity RFC's load table specifies. The + moved row is validated like a new one, and the old endpoint's cardinality is + released, as `validate.rs` already does for a moved `src`. +- **Cascade.** Deleting nodes of type `T` scans every edge table whose + endpoint set contains `T`. The filter is the type range on the packed + endpoint intersected with the deleted ids. Without the type half, deleting + `Person "alice"` would also delete edges to `Organization "alice"`. + +### Uniqueness, cardinality and keyed edges + +- `@card` counts out-degree per packed source over the one edge table. A + single table therefore holds cardinality across all endpoint types, which + is the rule real schemas ask for. +- `@unique` tuples that include a polymorphic `@src` or `@dst` include the + packed endpoint, so the same key to Person "alice" and to Organization + "alice" are two rows. +- A keyed edge's identity is its row's `__id`; `@key(@src, @dst, …)` is the + identity RFC's uniqueness rule over packed endpoints plus key properties. + There is no derived edge id to spell, and nothing changes when an endpoint + is widened, so generalizing a keyed or `@unique` edge is supported. +- Today's `@card` validator groups by the bare source id and reads deletions + from `node:{from_type}`. Both change for a polymorphic source; until they + do, `@card` on an interface source is refused (finding P6). +- An interface body constraint `@unique(p, …)` is an opt-in validation across + all members. It probes every member table, inside a write and on merge + deltas, and the proven fast-forward skip accounts for it. Lance key fencing + is per dataset, so cross-type uniqueness rests on read-set validation and + the exact graph-head precondition, never on a fence. Per-implementor `@key` + derived from an interface property keeps its per-table meaning. Before + publishing a schema change that introduces or widens an interface-wide + obligation, schema apply validates the complete prospective member universe + against the desired constraints. This covers added `implements`, inherited + membership and new body constraints, including existing rows without a write + delta. Duplicate tuples refuse the schema change atomically; publication + revalidates the captured graph and schema authority. + +### Schema migration + +| Step | Tier | Physical effect | +|---|---|---| +| Add interface, add interface inheritance | supported; validated if a uniqueness obligation is introduced or widened | none | +| Add an interface body `@unique` | validated, phase 3 | none; validate all prospective members | +| Rename interface (`@rename_from`) | supported | none | +| Add `implements` when the node already declares every inherited property compatibly | supported; validated if a uniqueness obligation is introduced or widened | none; satisfaction links change | +| Add `implements` with missing nullable properties | supported; validated if a uniqueness obligation is introduced or widened | today's add-property path for those properties | +| Generalize an endpoint from `C` to an interface or union containing `C` | supported, keyed and unkeyed edges alike | none; existing rows already carry `C` in their packed endpoints | +| Remove `implements`, remove an interface, narrow an endpoint | validated (the OG-MF-104 tier) | refused while any row's endpoint names a removed member, or a registered stored query depends on it | +| Fold an edge family into one polymorphic edge | explicit copy | rows of the named source edge types are copied into the new edge; the sources are tombstoned | + +Schema apply keeps its main-only, single-live-branch restriction and its one +manifest publication. Every step above fits that publication. + +### Wire, API and descriptor + +- Read results gain a `@type` column only where a query projects it. Abstract + bindings encode as ordinary flat columns. Arrow-json has no `Union` encoder, + and the descriptor refuses `Union`, so no result uses one. +- `ChangeEndpointsOutput`, export lines and load envelopes gain optional + `from_type`/`to_type`, present only for polymorphic endpoints. +- A structured, read-gated catalog route lists node types, interfaces with + their members, and edges with expanded endpoint sets. Clients stop + re-deriving implementor sets from `.pg` text. OpenAPI parity tests force + each consumer to classify it. +- Type-name filters on the change feed and export accept an interface name. + They expand it to members when the cursor is created and record the + expansion in the cursor. +- The operation descriptor lists every member type in `reads` and `writes`. + +### Policy + +Cedar has no type dimension today: its resources are graph, server and +cluster, and a request is action, branch and actor. An abstract binding is a +read of the same graph on the same branch, and an interface-wide mutation is a +`change`, so nothing changes. When per-type rules arrive (the planned +`predicate_for`), the planner expands each abstract binding to concrete types +before evaluating policy. `implements` never confers a permission, because a +`schema_apply` holder could otherwise widen read access. A traversal treats an +endpoint whose type is denied as filtered, never as a dangling reference. + +## Invariants + +- **1, respect the substrate.** Scans and searches use Lance plans per + dataset. Nothing duplicates a Lance primitive, and no storage changes. +- **2, one publication door.** Member-table effects of interface-wide + mutations and fold copies are staged as detached versions and published in + one `__manifest` CAS. No linear HEAD moves. +- **3 and 4, one coherent view, one publication.** Membership is resolved + once per invocation against the captured catalog. A fanned-out mutation + stages every member and publishes once. +- **6, identity from stable ids.** Endpoint types are `StableTypeId` halves + of packed references, never names, so renames rewrite nothing. Drop and + re-add mint a new id, and a validated step refuses to strand references. +- **7, derived state changes cost only.** Graph-index coverage and any later + interface projection change cost, never membership or results. +- **8, loud integrity failures.** Ambiguous direction, an unknown or + non-member endpoint type, an untyped polymorphic endpoint, cross-member + cardinality violations and an untyped expansion over a polymorphic edge are + typed refusals. +- **9, typed semantics.** Type sets, selections, type tests and pruning live + in AST, IR and plan structures. Type filters are structured predicates. +- **11, bounded resources.** Union arms run within the query's memory and + concurrency bounds. Polymorphic traversal shares the finite work budget. +- **12, one source of truth.** No identity registry or projection becomes + authoritative. The accepted schema plus each row's packed references are + the truth. + +Deny-list items reviewed: no ad-hoc `IN (...)` string generation, no eager +cross-product materialization in multi-hop execution (no per-type +sub-pipelines), no cost-blind plan choice, and no maintained parallel truth. + +## Compatibility and reversibility + +- Relative to the integer-identity baseline, a graph declaring no polymorphic + endpoint, interface inheritance or interface body constraint keeps + byte-identical IR, tables, load envelopes, exports and query results. +- Binaries that predate a feature refuse a graph that uses it, by unknown + feature name and by schema-hash mismatch. They never misread it. +- Wire changes are additive: optional endpoint types, a new catalog route, + and `@type` only when projected. Tower and other clients gate the grammar by + engine capability. +- Language and explain allocations follow [Queries](#queries). Union scans + require a new serialized-plan envelope version; its decoder preserves + compatibility for older saved plans that contain no union scan. +- Reverting the query surface refuses queries that use it. Reverting + polymorphic endpoints requires narrowing every polymorphic edge (validated) + and dropping the feature; no table is rewritten, since no storage changed. + A folded edge family can be split back only by a copy. + +## Alternatives + +| Alternative | Reason not selected | +|---|---| +| Do nothing | Families keep growing. Cardinality across a family cannot be declared, and alternation cannot combine different destinations. | +| A `StableTypeId` tag column on polymorphic edge sides, with the string key kept as identity (this RFC's first draft) | Two identity rules chosen per site by context, which the prototype got wrong four times (P1, P2, P4, P6); an unproven constant-column `Operation::Merge` to add the tag to existing tables; a rule for images from before the tag; a refusal to generalize keyed edges. The identity RFC puts the type into every endpoint and removes all four. | +| One table per hierarchy, with a `__type` discriminator and `extends` | Allows single inheritance only, while observed families are roles a type plays several of. Breaks one-type-one-table and the single-column primary key, and needs cross-table row moves that schema apply cannot publish. Its advantage, one index across types, can come later from a derived projection. | +| One physical edge table per endpoint pair (Kùzu's layout) | Table count grows with implementors times polymorphic edges, adding an implementor creates tables, and cross-member `@card` and `@unique` become cross-table validations. | +| Expand abstract bindings into per-type sub-pipelines | Multiplies pipelines across bindings, and gets mixed-type shortest distance wrong. The alternation RFC rejected the same shape. | +| Read properties missing on some members as null (Kùzu) | Wider, but makes absent and null indistinguishable in JSON and widens the static contract. The strict rule matches alternation; narrowing covers the rest. | +| Concrete supertypes (`node Employee extends Person`) | Raises key-scope and reclassification questions. Knowledge graphs model this by composition. Deferred. | +| Named unions as a declaration kind | Marker interfaces cover reuse. Inline unions cover closed endpoint sets. Deferred. | +| Resolve untyped polymorphic endpoints by probing every member table | Hides an N-table cost in every write, and turns colliding keys into errors at write time. Typed endpoints are the default, and probing is an unresolved question. | + +Prior art. Gel (EdgeDB) keeps a table per concrete type and compiles abstract +reads to a per-query `UNION ALL`, with `__type__` as a per-table constant. +That is the node side of this design. Kùzu and LadybugDB store +`(table_id, offset)` for every endpoint and partition relationship tables per +endpoint pair; TypeDB packs the type into the id; Hibernate `@Any` and Rails +polymorphic associations store an `(id, type)` pair. The identity RFC's packed +reference is the edge side. Neo4j's 2026 graph types use implied labels as +shared endpoint types. ISO GQL 2024 graph types allow one node type per +endpoint. PG-Schema models an edge type as a set of (source, edge, target) +triples, which is this design's semantics. + +## Evidence and tests + +### Code validation + +Before the first draft, 134 factual claims behind this design were checked +against OmniGraph `origin/main` (`420d7424`), the exact Lance 11.0.0, +DataFusion 54.0.0 and arrow-json 58.3.0 sources pinned in `Cargo.lock`, and +Tower. 106 held as written, 26 needed a caveat, and 2 were wrong; this RFC +records those original results below. They are author-reported historical +evidence, not rerun for this revision or proof of the new identity design. +The original test programs reported: + +- BTREE and Bitmap scalar indexes built on a `UInt64` column and served `=` + and `IN`. +- The same document scored 0.211 by BM25 in one dataset and 0.826 in another. +- `$x: Named` fails with T1, alternation with differing destinations fails + with T5, and `$x is Person` does not parse today. +- Tower's `.pg` parser drops `implements` after a bare-argument annotation, + and reads `A -> B | C` as `A -> B` without warning. + +The first draft's probes of a detached `Operation::Merge` adding a nullable +column, and of an Overwrite changing row ids and dropping indexes, supported a +storage design this revision no longer carries and are not evidence for the +integer-identity upgrade. + +### Prototype + +An end-to-end prototype on branch +[`proto/polymorphic-types`](https://github.com/ragnorc/omnigraph/tree/proto/polymorphic-types) +(at `74cb7a8f`; findings in its `PROTOTYPE_FINDINGS.md`) implemented +interface bindings, polymorphic edge endpoints and endpoint generalization +through every layer on the string-identity storage of the first draft. Its +tests use a Person "alice" and an Organization "alice" that share a key, so +any layer that matches endpoints on the key alone fails them. + +What held as designed, and carries over unchanged: the union scan conformed +each member table to the interface's columns plus `~node_type` through the +existing scan operator, with per-member pushdown; typed interner keys kept +colliding keys apart in forward, reverse, two-hop and negation traversals, and +in a `{1,3}` recursion over `RelatedTo: Named -> Named` across Person, +Organization and Note, with one scan per edge table and no per-pair members; +hydrating abstract destinations per member and joining on the typed key; +referential integrity grouped by type, and the type-predicated cascade; load +envelopes refusing a missing type on a multi-member side, a non-member type +and a type on a concrete side, and inferring the type for a one-member +interface. + +What the prototype found, and where each landed: + +| Finding | Layer | Effect without the fix | Where it lives now | +|---|---|---|---| +| P1 | loader | A tag among the properties is filled from user JSON. | Dissolved: there is no tag column. | +| P2 | lowering | Cycle closing on an interface silently dropped a row. | Identity RFC: the packed reference is the only identity. | +| P3 | typecheck | Narrowing had two sources of truth. | Typing and IR, above. | +| P4 | graph index | The index build collapses colliding keys into one node. | Graph index exclusion until phase 3, above. | +| P5 | migration | Generalizing a keyed edge would duplicate edges on the next keyed write. | Dissolved: edge identity is the row id. | +| P6 | validation | `@card` on an interface source groups different nodes and reads a nonexistent table. | Uniqueness, above. | +| P7 | migration | An all-null tag forces every read path to interpret null. | Dissolved: no tag column. | +| P8 | engine | An untyped expansion returned wrong rows with no error. | Planning and execution, above. | +| P9 | engine | Forcing CSR names an internal selection in its refusal. | Planning and execution, above. | +| P11 | typecheck | The ambiguity rule refused `Subject -> Subject` recursion. | Direction rule under Queries. | +| P12 | typecheck, engine | The multi-hop rule compared binding names in three places. | Typing and IR, above. | + +With the prototype's changes, the existing suites still passed: 465 compiler +tests, the planner suites, 706 engine tests, and all 275 GQT cases. + +### Tests to extend + +Following [the test map](../dev/testing.md), these existing owners are +extended: + +- **Compiler:** `schema/parser_tests.rs`, `catalog/tests.rs` (feature + derivation, byte-identical IR without the features), `query/parser_tests.rs`, + `query/typecheck_tests.rs`, `ir/lower_tests.rs`. +- **Planner:** plan round-trip tests for union scans. +- **Engine:** + - `tests/writes.rs` and `tests/validators.rs`: referential integrity by + type, cross-member `@card`, keyed polymorphic edges, Merge-load moves. + - Cascade: deleting `Person "alice"` leaves the edge to `Organization + "alice"`. + - Loader, export and change-feed tests: envelope keys. + - A plan that expands a polymorphic edge without endpoint types is refused + by `ExpandStep` validation, built directly rather than through typecheck. + - Cycle closing and negation over an interface binding with colliding keys. + - Direction: one-hop and recursive traversal over an edge with equal + endpoint sets (outgoing), and refusal over unequal overlapping sets unless + the other endpoint decides. + - A keyed polymorphic edge retains `__id` through an in-graph typed upsert + and restart. Export/load into another graph preserves its logical natural + key and relationships, with target ids governed by the identity RFC. + - Schema apply refuses duplicates introduced by direct or inherited + `implements`, and by a new body `@unique`, before publishing anything. + A pre-phase-3 binary refuses `interface-unique`, including empty interfaces. + - Generalizing a keyed edge keeps every existing row matchable by its key. + - Plan-replay tests: every member pin. +- **GQT:** cases that mirror the issue-659 suite. They cover union scans, + narrowing, projected type-test true/false values without row loss, negated + tests and disjunctions accepted by the other operand, `@type` ordering with + colliding keys, mixed-type + recursion, empty interfaces, historical refusal, `bm25` refusal and + forced-CSR refusal before phase 3, on the local filesystem and on + deterministic object storage. +- **DST:** concurrent writes to a polymorphic edge with cascades. +- **Reference engine:** refuses the new IR. +- **Tower:** catalog-route parity and grammar capability gating. + +Acceptance thresholds: + +- With full key-index coverage, a single-property key-equality lookup on an + N-member interface performs at most N scalar-index probes. Partial or absent + coverage scans uncovered data and produces the same rows. +- A fixture of one hub with leaves across three member types pins exact + traversal-work units for one hop and for `{1,3}`, with one-unit-below + refusal, as the alternation suite does. +- Generalizing an endpoint on a table of one million edges touches no data + file. + +### Defects found during the investigation + +- The investigation originally found a named cross-type multi-hop edge + silently running one hop. The reviewed compiler now refuses it with T5. + Polymorphic recursion is admitted only under the endpoint-set continuation + rule above; the old defect is not present in the reviewed base. +- Tower's insert-edge key pre-check finds the edge by `(name, from, to)`. For + a polymorphic edge it would skip the check, and a keyed insert would upsert. + Tower changes this before it enables polymorphic edges. + +## Rollout + +Each phase ships independently, and stopping after any phase leaves every +graph valid. Phases 1 and 2 require the identity RFC's storage upgrade on the +graph. + +1. **Phase 0, groundwork.** + - Validate interface properties at declaration. + - Warn when a redeclaration drops an interface constraint. + - Refuse new interface and node name collisions; support interface rename. + - Allow metadata-only `implements` additions, subject to schema validation + once interface-wide constraints exist. + - Make the three lowering sites handle type sets. +2. **Phase 1, abstract bindings.** + - Query surface: interface and union bindings, `@type`, type tests and + narrowing. + - Execution: union scans, interface-wide `nearest`, and interface-wide + `update`/`delete`. + - Interface inheritance behind `interface-inheritance`, and the catalog + route. +3. **Phase 2, polymorphic edges.** + - Endpoint declarations behind `polymorphic-endpoints`. + - Typed endpoint writes, validation, cascade, loads, export and + change-feed endpoint types. + - Budgeted indexed traversal. + - Generalization changes schema metadata only. Folding stages copied rows + and source-table retirement, then publishes them together. +4. **Phase 3, global index and uniqueness.** + - Global ordinal graph index (format v4) with admission at the cache + owner. + - Per-member statistics. + - Interface-scope `@unique` behind `interface-unique`, including full + prospective-universe validation on schema changes. Earlier phases refuse + its declarations and feature rather than accept unenforced constraints. +5. **Phase 4, extensions.** + - Covariant interface properties (enum refining String, non-null refining + nullable; physically identical). + - A top type (`$x: *`, `-> *`) that requires an anchor. + - Derived interface projections for cross-type full-text search. + - Downcast access `$x[Person].email`. + +`implementation` moves to `in-progress` with phase 0 and to `partial` when +phase 1 ships. + +## Unresolved questions + +Azim and Ragnor decide these extensions before their implementation; none is +required to admit phases 1 and 2. + +- **Bare-key endpoints.** Should `to: $slug` be accepted when it resolves in + exactly one member table, at the cost of probing every member, or must a + polymorphic endpoint always be typed? A derived per-interface key index + mapping a key to its packed reference would answer it without N probes; it + is not designed here. +- **Wide projection.** Should `return { $x }` offer a wide nullable struct + for multi-member sets, and what rule handles same-named properties of + different types? +- **Historical abstract bindings.** When can they be admitted? That needs + historical membership, which the captured contract does not reconstruct + today. +- **Interface-named policy rules.** Should they exist under per-type + authorization, compiled to the current member set and re-checked on every + schema apply? + +## Decision log + +2026-10-07: use type sets over per-concrete-type tables, with one union scan +per abstract binding and typed endpoint validation. + +2026-10-08: exclude polymorphic edges from the graph index until it has a +global ordinal space; refuse an untyped expansion and give named polymorphic +edges their own traversal policy. The former tag-column storage design is +superseded by the identity dependency below. + +2026-10-09: equal endpoint sets keep outgoing direction; only unequal +overlapping sets need disambiguation. Multi-hop continuation compares sets. + +2026-10-12: require integer identity before abstract bindings and polymorphic +endpoints. Packed references replace tag columns and their migrations, so +endpoint generalization is metadata-only even for keyed edges. Folding still +copies rows. Type tests retain Boolean expression semantics. Interface-wide +uniqueness ships with its own feature fence and schema-admission validation; +export/load preserves the identity RFC's logical-key contract across graphs. diff --git a/docs/rfcs/2026-10-12-distributed-sequence.md b/docs/rfcs/2026-10-12-distributed-sequence.md new file mode 100644 index 000000000..c3f21b735 --- /dev/null +++ b/docs/rfcs/2026-10-12-distributed-sequence.md @@ -0,0 +1,345 @@ +--- +rfc: "2026-10-12-distributed-sequence" +title: "Distributed sequence" +track: maintainer +status: draft +implementation: not-started +authors: + - azimafroozeh +created: 2026-10-12 +updated: 2026-10-12 +discussion: null +supersedes: [] +superseded_by: [] +blocked_on: [] +--- + +# RFC: Distributed sequence + +> A term set in ***bold italics*** is being defined at that exact spot. + +## Summary + +Every node table and every edge table gets a ***distributed sequence***, a +counter that hands out distinct 48-bit integers to any number of concurrent +writers through conditional reservations. The sequence is one small object +in the graph root, keyed by the stable table identity and incarnation, holding +the next unreserved number. A +writer reserves a block of consecutive numbers with one conditional write, +then allocates from that block in memory with no further I/O. Two writers +never hold overlapping blocks, so the numbers they hand out are distinct by +construction, whether the writers are two processes on one branch or two +branches that later merge. Numbers are unique within a table; they are never +required to be increasing across writers, and a block whose writer stops is +a gap, never a collision. + +The sequence frontier is independent of commit and never stored in +`__manifest`; the one existing publication door is unchanged. Ordinary writes +retain grants only in memory. The identity upgrade may record confirmed grants +in its frozen conversion plan so that a fenced attempt resumes the same row +assignments; that plan cannot grant new numbers or replace the frontier. +The first consumer is the integer node identity +RFC ([2026-10-12-integer-node-identity](2026-10-12-integer-node-identity.md)), +which takes every row's `__id` from its table's sequence. Later consumers may +include change-feed cursors, load batch numbers and WAL epochs; the sequence +does not know what its numbers name. + +## Motivation + +OmniGraph mints two kinds of identity today. Type identities come from one +counter, `Allocator` in `omnigraph-compiler/src/catalog/schema_ir.rs`, which +advances only when a schema is accepted, and schema apply refuses unless `main` +is the only live branch (`db/omnigraph/schema_apply.rs`, "schema apply requires +a graph with only main"). That counter is correct because exactly one writer +ever advances it. Row identities come from the user: a keyed node's `id` column +is `canonical_key_id` of its `@key` properties (`loader/mod.rs`), and an +unkeyed node's is a minted ULID string. + +The integer node identity RFC needs a third kind: a per-table integer handed +out on every insert, by every writer, on every branch. The type counter's rule, +one writer at a time, cannot apply to inserts. Branches exist so that writers +do not wait on each other, and within one branch writers prepare in parallel +and commit through one `__manifest` CAS. An id counter kept in a branch's +`__manifest` would be unique within that branch and collide with every other +branch at merge. A counter kept in one shared object and advanced on every +publish would put a second shared CAS on every write of every branch, which +is the contention the per-lineage design exists to avoid. + +Databases solved this long ago by handing out blocks. PostgreSQL's +`CREATE SEQUENCE ... CACHE n` gives each backend `n` values at once; Oracle's +cache does the same per instance; Hibernate's pooled optimizer and +Vitess sequences reserve a block from one shared row and allocate locally. +These designs accept gaps to amortize coordination across many allocations. +This RFC adopts that shape on the one coordination primitive OmniGraph has, +the conditional write on an object in the store, and gives it a name so that +later features reuse one tested component instead of growing their own +counters. + +## User and operational behavior + +Nothing user-facing changes. No GQ syntax, wire field, export line or CLI +command refers to a sequence. Operators see three things: + +- **One object per table lifetime under `__sequences/`.** Graph initialization, + added-table schema apply and the identity upgrade create it before enabling + number-consuming writes. Rename preserves it; drop and re-add use a new + identity. `omnigraph repair` reports missing or malformed sequence state. +- **Refusal when allocation state is unavailable.** After integer identity is + activated, a missing object is data loss, not an invitation to initialize a + counter. A new reservation refuses; an already-confirmed cache needs no + storage read to consume its grant and remains valid. Repair never resets the + counter. Backup and restore cover the whole graph + root, including sequences; restoring a counter independently is unsupported. +- **Gaps are normal.** A process that exits or crashes loses its unused + reserved numbers. Gaps carry no meaning and no tool reports them. + +## Design + +### The sequence object + +```text +/__sequences/{type_id}/{incarnation_id} +{ "next_id": 7168 } +``` + +Both path components are the decimal integers in the accepted `TableIdentity`, +not aliases or hashes of names. The graph root scopes that identity. Every +branch using the same table lifetime shares the object; rename changes no +path, and drop/re-add cannot inherit an old lifetime's numbers. + +`next_id` is the first number no reservation has consumed. A ***sequence block +reservation*** takes a positive size and follows this procedure: + +1. Read `next_id = H` and the object's version token. Refuse malformed state + or a value outside `0..=2^48`. +2. Compute `E = H + size` with checked arithmetic. Refuse exhaustion if + `E > 2^48`; there is no final short block. No conditional write occurs on + this refusal. `H == 2^48` is a valid exhausted state. +3. Write `{ "next_id": E }` with that token through `write_text_if_match` + (`omnigraph-core/src/storage.rs`). Only confirmed success grants `[H,E)`. + A definite precondition loss rereads and retries within the reservation + retry budget. An uncertain outcome grants nothing: abandon the candidate + range and return a typed storage error. An advanced counter cannot prove + who owns the range. A later ordinary reservation starts from a fresh read. + +On object stores, a true conditional put provides exclusion. On the local +filesystem, the adapter's compare-and-replace requires a caller-owned guard: +all reservation callers in one process share a guard keyed by canonical graph +root and stable table identity, held across read, compare and replacement. +Separate branch handles use the same guard. Local roots still support only +one writer process; this guard does not provide cross-process fencing. + +Creation has three owners: graph initialization for initial tables, schema +apply for newly added table lifetimes, and the offline identity upgrade for +tables whose accepted format predates integer identity. Each creates with +`next_id = 0` using create-if-absent, before the publication that activates the +consumer; an existing valid object is preserved, including on retry. The +upgrade may initialize populated legacy tables, and may find objects already +created by the sequence-only release. No owner recreates missing state for an +already-upgraded table. Aborted creation can leave an unused object; a retry +for the same table identity preserves it. Accepted stable identities are never +reset to reclaim gaps. + +### Supplied numbers + +Identity-preserving imports need an explicit reservation variant, not a +counter bump after staging. For one table, validate the complete supplied-id +set `S` before ordinary allocation for that table in the import. Require +distinct integers and `H <= id < 2^48` for every member. Conditionally advance +`next_id` to `max(S) + 1`; confirmed success grants exactly `S`, and skipped +numbers are consumed gaps. Empty sets do nothing. Conflict, synchronization +and uncertain-outcome rules are the same as block reservation. On conflict, +reread and recheck every id: the moved frontier can make the import invalid. + +An id below `H` is refused even if no current row uses it, because a live +cache, another branch or retained history may own it. Import preflight uses +bounded memory with spill or a typed resource refusal, not per-input-chunk +reservations that would reject an unsorted but valid set. Updates to existing +rows consume no new number; the identity RFC owns their matching rule. A +confirmed grant may be retained for retries in the same process. After loss +of that grant, the counter alone cannot authorize reusing the supplied ids; +there is no durable import-resume token in this RFC. + +### Session allocation + +A ***writer session*** is one process working on one branch. It holds one +***sequence cache*** per table it inserts into: the current block's end and a +cursor. Cache lookup uses the stable table identity within the canonical graph +root and branch session. Cursor allocation and refill have one synchronized +owner, so concurrent tasks sharing a session cannot emit an id twice. Within +that synchronization, allocation is a memory increment: + +```text +allocate(table): + if cursor == block_end: reserve the next block, set cursor and block_end + id = cursor; cursor += 1; return id +``` + +The cache is not durable. A restarted process is a new writer session, which +reserves a fresh block; the old block's remainder is a gap. No `__manifest` +row records an ordinary session's block or cursor. The cache also tracks the +next adaptive block size. The offline consumer upgrade's recorded grants are +the scoped exception above, owned by its pending conversion attempt. + +### Block size + +A session's first reservation on a table takes a block of `2^10`; each +exhaustion doubles the size up to `2^20`. A session that inserts one row +wastes about a thousand numbers; a bulk load reaches full-size blocks after +ten doublings, on reservation eleven. With 48 bits of number space, only +`2^48` numbers can be consumed over the table lifetime, including gaps and +import reservations. At a fixed size this permits `2^38` smallest blocks or +`2^28` largest blocks; it is not a bound on current live rows. The adaptive size exists so that +many short-lived sessions do not spend the space that bulk writers need. + +### Sequencing with the publish + +The sequence is consulted before staging, since the numbers it hands out are +written into the staged fragments. The publish CAS on `__manifest` is +unchanged and knows nothing about the sequence. The two conditional writes +touch different objects and never wait on each other: a session can hold a +block across many publishes, and a publish never reads `next_id`. Allocation +ownership does not validate a graph read set. Prepared writes keep their +`ExactGraphHead` precondition: head movement returns `ReadSetChanged` and a +retry captures a fresh snapshot, repeats key resolution and validation, and +restages when needed. The attempt may retain its confirmed number grant, but +must not reuse stale endpoint or uniqueness evidence. Abandoned numbers stay +consumed. + +### What the sequence does not do + +It does not order anything. Two sessions' numbers interleave arbitrarily, and +a row inserted today on one branch may carry a larger number than a row +inserted next year on another. Order of effects stays where it is, the graph +commit chain in `__manifest`. It does not identify the writer: nothing after +allocation recovers which session or branch minted a number, and nothing +needs to. + +## Invariants + +- **1, respect the substrate.** The sequence object is a plain object written + through the storage adapter's conditional put; no Lance mechanism is + duplicated or bypassed. +- **2, one publication door.** The sequence is not graph content. Reserving a + block publishes no graph rows, and no `__manifest` publication advances the + sequence frontier. The offline upgrade records confirmed grants only to + resume its frozen conversion plan. +- **5, crash convergence.** A crash between reservation and the first publish + that uses the block leaves `next_id` advanced and the block unused: a + gap, unreachable by construction, with nothing to reclaim. +- **11, bounded resources.** Each reservation attempt uses one conditional + write. Conflict retries have a finite budget, exhausted space refuses before + writing, and supplied-id preflight has bounded memory and spill. +- **12, one source of truth.** `next_id` is the durable frontier for future + reservations. A confirmed reservation owns its consumed range, and a cache + tracks allocation inside that grant. Losing a cache loses numbers; neither + a counter read nor a row-absence probe can recreate the grant. + +Deny-list items reviewed: the sequence is not a custom WAL, transaction manager +or storage primitive Lance owns, since Lance has no sequence; it is not a +process-local lock presented as distributed fencing, since the conditional +put is the fence and the local emulation keeps its existing boundary; it is +not a shadow copy of accepted state: the frontier alone admits new grants. +An upgrade plan records only grants already consumed by its fenced attempt. + +## Compatibility and reversibility + +- Before a number-consuming feature is activated, missing sequences are + permitted and reads are unchanged. Initial creation or the identity upgrade + prepares them as specified above. After activation, missing state refuses + allocation and is never silently initialized. +- A binary from before this RFC ignores `__sequences/`. With no consumer, + reverting the unused facility changes no rows. Once a consumer is active, + its feature and storage-format fences govern compatibility; sequence state + must be retained with its identities. +- The object format carries no version field. A later change to its encoding + uses a new object prefix, never a reinterpretation of `next_id`. + +## Alternatives + +| Alternative | Reason not selected | +|---|---| +| No sequence: a per-table counter in each branch's `__manifest`, advanced in the publish CAS | The minus-one-mechanism design. Unique within a branch, and two branches both hand out the same numbers for different rows; merge would renumber one side and rewrite every edge referencing it. The failing scenario is any two branches that both insert into one table, which is the documented purpose of branches | +| One shared counter advanced on every publish | Correct, and every write on every branch takes a second CAS on one hot object. Branch writers would contend with each other on every publish, which the per-lineage `__manifest` design exists to prevent | +| Blocks granted per branch at `branch create`, cursor in the branch's `__manifest` | Correct for one writer per branch. Two writers on one branch read the same cursor, stage fragments with the same numbers, and the loser of the publish CAS must re-stage with fresh numbers. Per-session blocks keep allocation independent of publication retries, while each retry still revalidates its graph read set | +| Random 48-bit numbers | No coordination at all, and birthday collisions become likely around sixteen million rows in one table. Not acceptable for an identity | +| Static writer prefix, `writer_id << 40` | A fixed cap on writers and a fixed share of the space per writer. The block design is this with dynamic sizing and no registry of writers | +| Lance row addresses or move-stable row ids as the numbers | Owned by Lance, renumbered by compaction or reassigned on update and on merge, allocated per lineage. Not a logical identity, which is invariant 1's line | + +Precedent audit. The nearest in-repo pattern is `Allocator` for type ids: a +counter whose correctness rests on a single-writer rule. This RFC diverges by +partitioning the number space because inserts have many writers, and keeps +the one-writer rule only at the single point where the space is partitioned, +the conditional put on `next_id`. The branch-ref carrier is the other +graph-wide serialized object; the sequence is kept out of it so that a +reservation never contends with `branch create`, and so that one table's +reservations never contend with another's. + +## Evidence and tests + +Following [the testing map](../dev/testing.md): + +- **In-source `omnigraph-core` tests** for the object format, the block + arithmetic and mixed-session adaptive sizes; checked overflow, malformed + state, the last valid id and the first refused reservation. +- **Engine `writes.rs`:** N concurrent sessions reserving from one table + receive disjoint blocks; a session that exhausts a block continues with the + next; a table whose sequence object is missing refuses the write with the + typed error. Two branch handles on one local root must share the reservation + guard; concurrent tasks in one session must share cursor/refill ownership. +- **`integration/branching.rs`:** two branches inserting into one table and + merging produce no equal numbers. +- **`failpoints.rs` and `detached_commit_matrix.rs`:** a crash after the + reservation write and before the first publish leaves `next_id` + advanced, the block unused, and the next session on a fresh block; a crash + after publish leaves the published rows' numbers intact and the remainder + a gap. Inject a successful conditional put with a lost acknowledgement; the + failed caller emits no candidate id. Test supplied ids below a live cache + frontier, unsorted supplied ids, and import reservation conflicts. +- **`forbidden_apis.rs`:** no code path writes `__sequences/` except the + reservation, graph initialization, added-table schema apply and the legacy + identity upgrade; no path resets existing state. Test rename, drop/re-add, + creation retry, pre-created upgrade objects and missing upgraded state. +- **DST:** concurrent writers across branches with crash injection at the + reservation; the oracle is "every number in every table appears in at most + one row on every branch and after every merge". + +Acceptance thresholds: each granted block has one successful conditional write +and cached allocation has no per-row I/O; an uncontended run of one million +inserts in one session performs at most +eleven reservations. Use a `sequence_bulk_million` fixture with one table, +one session and exactly 1,000,000 allocations; a `sequence_short_sessions` +fixture uses 1,000 sessions allocating one id each. Record reads, conditional +write attempts, successful reservations and unused numbers separately. + +Surfaces surveyed: `write_text_if_match` and its local emulation +(`omnigraph-core/src/storage.rs`), the publisher's retry budget, the schema +apply branch guards, PostgreSQL's sequence cache semantics and Hibernate's +pooled optimizer as the prior art for block reservation. + +## Rollout + +1. The sequence object, reservation, cache and refusal, with no consumer. + Initialization and added-table schema apply create objects for new table + lifetimes. Ships alone; nothing reads the numbers yet. +2. The integer node identity RFC becomes the first consumer and creates + sequence objects for existing tables in its storage upgrade. + +`implementation` moves to `in-progress` with phase 1 and to `complete` when +the consumer RFC's upgrade creates the objects for existing tables. + +## Unresolved questions + +- **Block size constants.** `2^10` first block, doubling to `2^20`, are + chosen from the arithmetic above and not from a measurement. The + implementation measures both named fixtures above. Azim selects the + constants from reservation count and unused-number measurements; changing + the sizes does not change the ownership protocol. + +## Decision log + +2026-10-12: use one graph-wide sequence per stable table lifetime and adaptive +writer-session reservations. Persist the next unreserved number so unequal +block sizes remain disjoint. Keep allocation separate from graph publication; +local reservation requires the adapter's caller guard. Supplied ids must be +above the reserved frontier, and exhaustion refuses before reservation. diff --git a/docs/rfcs/2026-10-12-integer-node-identity.md b/docs/rfcs/2026-10-12-integer-node-identity.md new file mode 100644 index 000000000..6945bb2cf --- /dev/null +++ b/docs/rfcs/2026-10-12-integer-node-identity.md @@ -0,0 +1,591 @@ +--- +rfc: "2026-10-12-integer-node-identity" +title: "Integer node identity" +track: maintainer +status: draft +implementation: not-started +authors: + - azimafroozeh +created: 2026-10-12 +updated: 2026-10-12 +discussion: null +supersedes: [] +superseded_by: [] +blocked_on: + - "2026-10-12-distributed-sequence" +--- + +# RFC: Integer node identity + +> A term set in ***bold italics*** is being defined at that exact spot. + +## Summary + +Every node and every edge row gets its identity from the engine instead of +from the user. The identity column `__id` (`id` on legacy-vintage graphs) +changes from `Utf8` to `UInt64`, filled from the table's distributed sequence +([RFC](2026-10-12-distributed-sequence.md)). An edge endpoint stores one +`UInt64`, the ***packed node reference***: the endpoint type's `StableTypeId` +in the high 16 bits and the node's `__id` in the low 48. A packed reference +names a node by itself, with no table, declaration or binding type to tell +the reader where to look, so an edge whose declaration allows several +endpoint types needs nothing beyond the column every edge already has. + +The `@key` is unchanged as the way users address a node: unique within its +type, required, the value users write in `insert`, `where` and load +envelopes. It stops being the stored identity. A write that names a node by +key resolves it to its `__id` against the captured snapshot, using available +index coverage plus scans of uncovered data. The engine compares, joins and +interns packed references; export renders them as public addresses. Nothing +in GQ, in results or in exports shows a raw integer for a keyed node. + +What does not change: one Lance dataset per node or edge type, the one +`__manifest` publication door, three-way merge and keyed-node addressing. +The accepted schema gains the identity feature and packing contract. Unkeyed +node handles and every edge handle change, including on monomorphic graphs; +the wire rules below define that compatibility boundary. + +## Motivation + +A node's stored identity today is a string: `canonical_key_id` of its `@key` +properties (`loader/mod.rs`) for a keyed type, a minted ULID for an unkeyed +one. The string is unique inside one table and nowhere else: `Person "acme"` +and `Organization "acme"` are two legal nodes. An edge column holds the bare +string and relies on the edge declaration to say which table it means. That +works as long as every edge side names exactly one type, and it is why the +polymorphic types RFC ([2026-10-07-polymorphic-types](2026-10-07-polymorphic-types.md)) +had to add a tag column, a metadata-only migration to put the tag on existing +tables, a rule for images from before the tag, and a refusal to generalize +keyed edges whose ids were derived without a type. Its prototype found four +defects (P1, P2, P4, P6) that were all one shape: a code path compared bare +ids where the pair was needed. + +The string also sits in the hot path. Every join, dedup, cycle check, +anti-join and traversal frontier compares and hashes strings, and the graph +index interns them to dense `u32` per type (`TypeIndex::get_or_insert`, +`graph_index/mod.rs`) before a CSR can exist. Kùzu and LadybugDB traverse on +`(table_id, offset)` pairs of two integers; Gel compiles abstract reads over +per-type tables that share one id space; Hibernate and Rails store `(id, +type)` for a reference that may point into several tables. Every system with +engine-minted identity made the same move: a reference that carries its own +type, in fixed width. + +An issue or local refactor cannot do this. It changes the identity column's +type on every table, the endpoint columns of every edge table, the loader's +id derivation, merge's row matching, the export format and the graph index, +and it introduces the first consumer of the distributed sequence. + +## User and operational behavior + +### What users see + +- `insert Person { slug: "acme" }`, `update Person set ... where slug = "acme"`, + `delete Person where slug = "acme"` and load envelopes keyed by `slug` work + as today. The key is the address. +- `$p.@id` for a keyed node is the key value, as today. For an unkeyed node it + is `Type:number`, for example `Note:1048576`, where `number` is the low + 48-bit row id and `Type` is the current canonical name. The internal pair is + stable; the spelling changes on rename and the old name is not an alias. + Handles are graph-local and never prove the origin of imported data. +- `$e.@id` for every edge, keyed or unkeyed, is the String handle + `EdgeType:number`. It replaces both derived keyed-edge ids and ULIDs. + `$e.@src` and `$e.@dst` render the endpoint node's public address: its key + for a keyed node, or `Type:number` for an unkeyed one. Filtering these + pseudo-properties resolves the address against the captured schema and + compares typed identities. A multi-member endpoint requires the typed form + from the polymorphic RFC; a bare key cannot select an arbitrary member. +- Result ordering tie-breaks, `@type` ordering and dedup are unchanged in + meaning. Rows that tie on every sort key now tie-break on the packed + reference, so a result's order within ties may differ from the string + order of today. +- A keyed node's key may be changed by an ordinary `update` once this RFC + ships, because nothing stored points at the key. Today a key change is a + delete plus an insert of a new node. + +### What operators see + +- A storage upgrade, run once per graph by the operator with + `omnigraph upgrade`, under the same rule as the system-column upgrade: it + refuses while any branch other than `main` exists and converts the current + snapshot of every node and edge table. The offline upgrade protocol fences + the root during conversion and activates all converted tables together. + Before conversion, the new binary can read the old identity format but + refuses ordinary writes; while upgrade is pending, normal open refuses. +- Natural-key indexes accelerate address resolution; surrogate-id indexes + accelerate row lookup. Neither is an admission requirement. Missing or + partial coverage uses the ordinary scan fallback, and expensive index + building remains explicit optimize/reconcile work. +- Export writes keyed nodes and endpoints by key, as today, and unkeyed ones + by `Type:number`. An optional `__id` field exists in the format for the + loader's supplied-number reservation and is written for unkeyed nodes and + unkeyed edges. Keyed rows are matched by natural key on import. + +### Failure posture + +- A write naming a key that resolves to no node fails as today, a typed + integrity error. +- A new reservation whose sequence object is missing is refused with the + sequence RFC's typed error; previously confirmed grants remain valid without + per-row storage checks. Only a legacy-format upgrade may initialize + missing state; an upgraded graph must not reset its counter. +- A packed reference whose type half is not a type in the accepted schema, or + whose node is absent, is a loud integrity failure on read, never a + placeholder. + +## Design + +### Columns + +| Column | Today | After | +|---|---|---| +| `__id` on a node table | `Utf8`, non-null, `canonical_key_id(key)` or a ULID string | `UInt64`, non-null, from the table's sequence | +| `__id` on an edge table | `Utf8`, non-null, derived from the key tuple or minted | `UInt64`, non-null, from the edge table's sequence | +| `__src`, `__dst` on an edge table | `Utf8`, the endpoint's string id | `UInt64`, the packed reference of the endpoint | +| `@key` properties | Unique within the type, the identity | Unique within the type, indexed, no longer the identity | + +`physical_table_schema` (`db/omnigraph.rs`) keeps marking the identity column +with Lance's unenforced primary-key metadata and the `SystemFieldRole` set +`Id`, `Src`, `Dst` (`catalog/schema_ir.rs`) is unchanged; only the Arrow types +behind the roles change. The system column spellings stay a function of the +graph's `SchemaIR::ir_version`, so a legacy-vintage graph's `id`, `src` and +`dst` carry the new types under their old names. + +### The packed node reference + +```text +packed = (StableTypeId << 48) | __id as UInt64 +type = packed >> 48 16 bits, numeric type id <= 65535 +id = packed & (2^48 - 1) 48 bits, allocated id < 2^48 +``` + +`StableTypeId` comes from `Allocator` in `catalog/schema_ir.rs`, a counter +advanced only at schema acceptance. That allocator also issues property and +table-incarnation ids: 16 bits is a bound on a numeric type id, not a promise +of 65535 live types. Schema acceptance and upgrade preflight reject any type +id above 65535; they never renumber stable identities. Properties and +incarnations keep their existing widths but consume the shared allocator, so +they can make a later type allocation exceed the bound. A separate allocator +is outside this RFC. The low-half capacity counts all reserved numbers, +including gaps, not just live rows. Both halves are rename-stable (invariant +6), so a type rename rewrites no edge row; public name-bearing handles render +with the new canonical name. + +One `UInt64` was chosen over two columns and over 16-byte forms for three +reasons: Lance's BTREE and Bitmap scalar indexes and `IN (...)` pushdown on +`UInt64` are proven today; a type-only predicate such as cascade's "every edge +to a `Person`" is one range scan, `packed BETWEEN type << 48 AND +((type << 48) | (2^48 - 1))`; and the pairing of type and id is in the value, not +in a convention two columns must keep. The 16/48 split is a per-graph constant +recorded in the accepted `SchemaIR`, never a code constant. + +### `NodeRef` + +Upgraded-format execution gains one type, +`NodeRef { type_id: StableTypeId, id: u64 }`, with +checked construction, `pack()` and `unpack()`. Construction rejects a type +id above 65535 or row id at least `2^48` before shifting. Unpacking validates +the type against the captured schema; row existence remains an integrity +check. Endpoint-type range bounds use the masked maximum above, including +for type 65535, without shifting 65536 into a `u64`. Traversal +frontiers, visited sets, the graph index interner, dedup, cycle closing, +anti-join keys, rank-fusion identity, hydration lookups, cascade filters, the +change feed's endpoint images and export all take a `NodeRef`. A bare `u64` or +a bare string is not accepted where a node is meant, so the condition "pair +where a set has more than one member, bare id otherwise" has no place to +exist. + +### Writes + +A write that names a node by key, `to: "acme"` or `where slug = "acme"`, +resolves the key to its `__id` before staging. The natural-key index answers +key predicates; an index on `__id` answers surrogate-id predicates. Resolution +batches keys by type with structured predicates, using covered index results +and scans of uncovered fragments at the same snapshot. Absent coverage changes +cost only. A key that resolves to nothing is the existing integrity error. +New rows take their `__id` from the sequence cache. A moved graph head returns +`ReadSetChanged` under `ExactGraphHead`; retries capture a fresh snapshot, +resolve keys and validate again, and restage where required. A retained +confirmed allocation can supply the same ids, but cannot validate stale reads. + +Referential integrity, cardinality and uniqueness validation run on packed +references. A `@card` bound is counted per `(src packed)` over one edge table. +`@unique` tuples that include `@src` or `@dst` include the packed value. + +### Keyed edges + +An edge's identity is its row's `__id` from the edge table's sequence. +`@key(@src, @dst, ...)` on an edge is a uniqueness rule, matched on write +over the packed endpoint columns plus the declared key properties, which are +indexed. No edge id is derived from a string. `canonical_key_id` keeps one +job: encoding a node's composite natural key for its unique index. + +### Merge + +Three-way matching first aligns rows inherited from the merge base by stable +`__id` and recorded merge correspondence (defined below), including rows whose +natural keys changed. Divergent key updates and +delete/update cases use the ordinary three-way conflict rules. Only rows +absent from the base on both sides are candidates for ***natural key match***: +independent inserts with equal keys are paired despite unequal surrogate ids. +The merge then performs ***surrogate key remapping***, replacing references to +the losing inserted node with references to the survivor. + +For a non-fast-forward merge, the order is normative: + +1. Resolve correspondence from the three captured snapshots as described + below, then classify inherited node rows by that stable identity. Pair + independent keyed-node + inserts by natural key. The target's `__id` survives; equal payloads + coalesce and property disagreement follows the existing conflict policy. + Unkeyed inserts remain separate nodes. +2. Build `{losing packed -> surviving packed}` from those node pairs. + Normalize source endpoint references through it across all source rows + contributing to the candidate result, including otherwise unchanged edge + tables. Target references are already in the surviving namespace. +3. Match inherited edge rows by stable identity and correspondence. For + independent keyed-edge + inserts, compare natural-key tuples only AFTER endpoint normalization. + Equal keys and payloads coalesce under the target edge's `__id`; differing + non-key properties follow the existing insert conflict policy. Independent + unkeyed edges keep their distinct ids. +4. Validate the complete candidate result, including node and edge keys, + referential integrity and cardinality. A key reused by a different inherited + row is a uniqueness conflict, not permission to fuse their identities. + Publish the rows and new correspondence together under the captured merge + authority. + +Fast-forward adopts the source snapshot, including its correspondence pin and +surrogate representatives, verbatim. It neither applies target-survivor +selection nor remaps rows. A coalescing merge, including a later fast-forward, +can therefore change which surrogate represents a keyed row and its edge +handle; ordinary updates and restarts preserve the chosen id. A key update +itself never changes that id. + +#### Correspondence across repeated merges + +A first merge can coalesce source id `S` into target id `T` while the source +branch still stores `S`. Its next merge may use the old source snapshot as +the base. Comparing raw ids alone would then mistake `T` for an unrelated +insert, and matching current natural keys would fail after a key update. + +Each coalescing merge therefore records ***merge correspondence***: a relation +between row ids treated as one logical row, scoped by stable table identity +and incarnation, for nodes and edges. It is authoritative merge state in one +internal Lance dataset. Its non-null `UInt64` columns are `type_id`, +`incarnation_id`, `row_id` and `class_id`; `(type_id, incarnation_id, row_id)` +is unique. Every member of a non-singleton class has a row, and `class_id` is +the least row id in that class, not necessarily its current live surrogate. +Absent rows represent singleton classes only when the captured correspondence +snapshot is complete. An absent optional pin means the relation is empty; a +present but unreadable pin is an integrity refusal. + +An optional correspondence-table pin is added to accepted graph state and +retained in its history records. It uses the existing detached-table staging, +version-pinning and collection protocol; the integer-identity format fences +readers that do not know the field. A branch fork inherits the pin and an +ordinary write carries it unchanged. No mutable root-side registry or full +ancestry reconstruction is needed. Ordinary queries never redirect old ids +through this relation. + +For identities participating in classification, look up classes in the +captured base, source and target correspondence tables. Expand only classes +intersecting that working set, taking their union across the three images. +Use structured `row_id`/`class_id` lookups with index-plus-uncovered-scan +fallback and the existing merge resource budgets. No per-commit history walk +is performed. Two row ids in one class align for three-way comparison; +absence from a row table remains deletion, not a row synthesized by the map. +An input containing two live rows in one of its own recorded classes is an +integrity failure. If combining individually valid relations places two live +rows from one input in the same class, return a typed correspondence merge +conflict; do not choose a survivor or call the stored input corrupt. +Property comparison ignores surrogate-id differences. A class +with a target row uses its id in the result; otherwise a retained source row +keeps its id. Normalize endpoint references in every comparison image into +that selected namespace before comparing edges. + +The result carries the union of the source and target relations, including +correspondence for deleted rows that a later merge can encounter. Seed class +expansion with every source correspondence-delta member and new coalescence, +as well as row-classification candidates. Repeatedly fetch intersecting classes +from all three pins until no member is added, including deleted-row ids. +Union the equivalences, assign the minimum member to every row in each affected +class, and stage target inserts/updates through the ordinary detached-table +machinery. This is set union, not per-row three-way conflict selection: +`{1,2}` and `{2,3}` become `{1,2,3}` even though row 2 had different stored +`class_id` values. Unaffected classes retain their versions; +rewriting the whole relation on each merge is not required. Publish its pin +atomically with the result's row tables. Allocation, lookup, closure and +staging are charged to existing memory/work budgets; a refusal publishes +nothing. The upgrade starts with an empty relation. Retention protects pinned +versions as it protects all other accepted table versions. + +### Export and load + +The existing envelope `id` field carries a public address, not a raw numeric +`__id`. Export writes a keyed node by key and an unkeyed node by `Type:number`; +every edge's `id` is `EdgeType:number`. Edge `from`/`to` carry endpoint node +addresses, with `from_type`/`to_type` only when that endpoint allows multiple +types. Optional numeric `__id` is emitted only for unkeyed nodes and unkeyed +edges. It denotes the low 48 bits; the target catalog supplies the type id. +Source and target graphs need not have the same stable type ids. + +Load follows these matching rules, at one captured target snapshot: + +| Row kind | Matching and allocation | +|---|---| +| Keyed node | Match its natural key. An admitted upsert retains the target id; a new row mints an id. A supplied public `id` must agree with the key. Numeric `__id` is refused in this version. | +| Keyed edge | Resolve endpoints first, then match the declared natural key. An admitted upsert retains the target id; a new row mints an id. The exported public `id` is descriptive and does not select the target row. Numeric `__id` is refused in this version. | +| Unkeyed node or edge with numeric `__id` | This requests a fresh identity-preserving import, never an upsert. Reserve the complete supplied-id population by the sequence RFC's supplied-number procedure. An existing row or an id below the reserved frontier refuses, even in Merge mode. If public `id` is present, its type and number must agree. | +| Unkeyed node or edge with only public `id` | In Merge mode it addresses an existing target row; absence refuses. No number is reserved. Other modes refuse this form rather than guess whether the handle is foreign. An unkeyed edge may change endpoints, subject to the usual validation. | +| Unkeyed node or edge with neither field | Mint a new id from the sequence. | + +An export containing unkeyed rows therefore restores them into a target only +when its supplied ids pass the reserved-frontier check. Equal numeric handles +from another graph are never silently interpreted as an upsert. To update an +existing unkeyed row, a caller deliberately uses Merge with `id` and omits +`__id`. Failed imports may leave reserved gaps; losing the confirmed grant +prevents retrying the same supplied ids. Arbitrary id-remapping imports and +durable import resumption are outside this RFC. + +Endpoint resolution uses target natural keys or the imported/unkeyed row +mapping, never source packed values. Node rows are resolved before edge rows. +Across graphs, keyed rows preserve logical keys and relationships but may get +different `__id` values. Within a graph, upserts and restarts retain row ids. + +### Graph index and traversal + +`TypeIndex` interns packed references instead of strings; the CSR build's +string intern pass becomes an integer hash map. The per-type dense `u32` +spaces remain derived state (invariant 7). A later RFC may place them at +fixed offsets in one global ordinal space so a polymorphic edge has one CSR; +this RFC does not. + +### Storage upgrade + +This is a new route in the existing offline upgrade protocol +(`db/upgrade.rs`, [versioning](../dev/versioning.md)), with a new supported +source/target stamp and protocol version allocated at implementation. It +extends `UpgradeIntent` and the pending-upgrade fence; it does not add a +second recovery authority. The system-column upgrade supplies the main-only +guard, but its column rename alone does not solve row-identity conversion. + +1. With all writer and maintenance processes stopped, require main to be the + only live branch. Capture its exact source snapshot, schema contract and + table pins. Preflight type-id bounds, unique old stored ids, endpoint + resolution, row counts and resource limits before converting any data. +2. Initialize or reuse each table's sequence by its stable identity. Reserve + enough numbers for its pinned row count, with checked bounds. Bind those + confirmed ranges, exact source pins and schema to the immutable conversion + plan in `UpgradeIntent`; fence main with that intent under the exact source + authority. A source change refuses. Reservations lost before the intent + records them remain gaps. Empty tables need no range. +3. For each table, sort the pinned source's old stored ids by a fixed binary + UTF-8 order. Assign ordinal `n` the nth number in that table's recorded + ranges. This maps every old identity, including unkeyed ULIDs, to its new + id independently of physical row order. Node maps produce `NodeRef`s; + convert edge endpoints through those maps. Build transient maps with + bounded memory and spill, or refuse the configured resource limit. +4. Stage detached table versions and checkpoint their exact pins as progress + receipts bound to that intent and source pin. The frozen allocation plan + never changes. On resume, reconstruct maps from the pinned source and + recorded ranges, validate receipts, reuse completed versions and restage + only incomplete tables with the same assignments. A corrupt receipt is a + refusal, not permission to guess a mapping. The pending attempt pins its + source and reusable staged versions; collection must retain them until + activation or explicit recovery. Ordinary open and maintenance refuse + while the fence is pending. +5. Validate all staged rows and references, then activate the converted table + set, schema feature and target stamp together through one `__manifest` + publication under the pending attempt's exact authority. Control commits + for fencing and progress do not expose a partly converted graph. Recovery + checks the receipt after an uncertain activation outcome before retrying. + +### Retained history + +Only the current snapshot is converted. Retained source table versions keep +their string ids and are decoded using each snapshot's captured contract and +storage stamp. Concrete historical queries and exports remain available in +their old identity representation; the upgrade does not retroactively assign +integer handles to old history. A change-feed cursor reaching the identity +upgrade returns a typed format-boundary refusal before emitting that +transition. It does not pair old strings with new integers or report every row +as changed. The caller takes a fresh export of the upgraded snapshot and +starts a cursor after that snapshot. A feed wholly within either format uses +each image's captured contract for endpoint rendering. + +A branch cannot be created at a pre-identity snapshot, and merge refuses any +source, target or merge base at the old identity format. Writes to such a +snapshot refuse; they do not perform an implicit upgrade. The offline route +above upgrades current main only. Historical abstract bindings remain the +polymorphic RFC's explicit refusal. These admission checks apply before +staging and leave read-only history accessible. + +## Invariants + +- **1, respect the substrate.** `__id` and the endpoints are ordinary Lance + `UInt64` columns with ordinary scalar indexes. No Lance row address or + stable row id is used as identity. +- **2, one publication door.** The upgrade and every write publish through + `__manifest` as today. +- **3 and 4, one coherent view, one publication.** Key resolution reads the + captured snapshot; a write stages all its tables and publishes once. +- **5, crash convergence.** Staged upgrade versions are hidden from normal + queries until activation, while pending-attempt pins protect recovery; + a lost block is a gap. +- **6, identity from stable ids.** Both halves of a packed reference survive + renames; drop and re-add mint a new type id, and the upgrade refuses to + strand references. +- **7, derived state changes cost only.** The key index and the graph index + are derived; missing or incomplete coverage uses scans over the captured + snapshot and changes cost, not write admission or results. +- **8, loud integrity failures.** An unresolvable key, an unknown type half, + an absent node and a colliding imported id are typed refusals. +- **9, typed semantics.** `NodeRef` is the identity carrier in upgraded-format + execution. Legacy decoding keeps the old string representation for retained + historical reads and never admits it into upgraded-format writes. +- **12, one source of truth.** The accepted schema plus each row's `__id` are + the current identity truth. The accepted correspondence pin records + coalesced-row equivalence solely for three-way matching; it does not redirect + ordinary reads. There is no id-to-type registry, because the type is in the + reference. + +Deny-list items reviewed: no ad-hoc `IN (...)` string generation (key probes +are structured predicates), no maintained parallel truth (the key index is +derived, the sequence is the sole allocation authority), no process-local +lock presented as fencing (the sequence RFC's conditional put is the fence). + +## Compatibility and reversibility + +- A graph at the old format is read by the new binary and refuses writes + until upgraded. An old binary refuses a graph at the new format by + schema-hash mismatch and a feature name, `integer-identity`, in + `required_features`. +- Wire: `@id` of a keyed node is unchanged. `@id` of an unkeyed node changes + from a ULID string to `Type:number`; every edge `@id` becomes + `EdgeType:number`. Name-bearing handles change spelling on rename. Export + gains optional numeric `__id` for unkeyed rows and conditional + `from_type`/`to_type`; import matching follows the table above. +- The published rule "a keyed node's id is its key" becomes "a keyed node is + addressed by its key". Documentation and Tower change the sentence; no + client that addresses nodes by key changes code. +- Reverting after the upgrade is a second rewrite of every table, deriving + string ids from keys and minting ULIDs for unkeyed nodes. It is a format + downgrade, not a toggle. + +## Alternatives + +| Alternative | Reason not selected | +|---|---| +| Keep the string key as identity and add a type tag column on polymorphic edge sides (the polymorphic RFC's original storage design) | The minus-one-mechanism design. Two identity rules, bare id and pair, chosen per site by context; the prototype's P1, P2, P4 and P6 are four sites that chose wrong. Strings stay in the hot path. The tag's migration needs an unproven constant-column `Operation::Merge`, a historical-image rule and a keyed-edge refusal | +| Globally unique user keys | Breaks two legal nodes with the same key in two types; rewrites every key users chose | +| A type-name prefix in the id string, `"Person:acme"` | Stringly typed; a rename rewrites every row; every consumer parses | +| Two columns per endpoint, type and id | Same information, pairing by convention; a type-only filter needs a second indexed column; nothing it does that one packed value does not | +| `FixedSizeBinary(16)` or `Decimal128` per endpoint | 64 bits of id space, and Lance scalar indexes on those types need a probe that `UInt64` does not | +| Hash of the key as the low 48 bits | No allocator and branch-safe by construction, but no insertion order or locality and a collision rule at about sixteen million rows per type | +| Lance row address or move-stable row id as the low half | Renumbered by compaction or reassigned on update and merge, allocated per lineage, owned by Lance | +| Kùzu's dense per-table offset | Needs one allocator per table; branches are many allocators. The distributed sequence is this offset made branch-safe | +| Discard the remap after each merge | After coalescing source id S into target id T, a later merge can have S in its base and T in its target. Mutable keys prevent recovering correspondence by current key. A version-pinned relation retains it under the existing commit authority without replaying full history | + +Precedent audit. `StableTypeId` is the in-repo pattern: an engine-minted +integer identity that survives renames, with the name as an alias. This RFC +extends the same pattern from types to rows, and diverges only in the +allocator, because rows have many concurrent writers and types have one. The +system-column upgrade is the precedent for the storage upgrade's guard and +shape. + +## Evidence and tests + +Following [the testing map](../dev/testing.md): + +- **Compiler:** `catalog/tests.rs` for the `StableTypeId` cap and the + `integer-identity` feature derivation; `query/typecheck_tests.rs` for `@id` + typing on keyed and unkeyed nodes, all edge handles and endpoint filters. + Include a graph with few types but property/incarnation churn above the + numeric type limit; upgrade must refuse without renumbering. +- **Engine:** + - `writes.rs` and `integration/validators.rs`: key resolution, batched + probes with full, partial and absent index coverage, unresolvable keys, + `@card` per packed source, `@unique` with packed endpoints, keyed-edge + matching, and a head move that deletes a resolved endpoint before retry. + - `integration/merge_truth_table.rs` and `integration/merge_fast_forward.rs`: + the insert-insert case with one edge on each side, resulting in one node, + both edges, one `@id`; the remap over an edge table with no other changes; + property disagreement still a conflict; unkeyed double inserts stay two + nodes. Add the same keyed edge on both branches to independently inserted + equal-key nodes, divergent inherited key updates, key reuse and key + uniqueness conflicts. Repeat merges after coalescing nodes and keyed edges, + after a key update, after deleting a survivor, and through a third branch; + settle history and restart between merges. Cover missing correspondence, + transitive classes, union-only correspondence deltas for deleted rows, + incompatible but individually valid classes, and resource-budget refusal. + - Cascade: deleting `Person "acme"` leaves the edge to `Organization "acme"`. + - `integration/export.rs`: keyed logical round trips and unkeyed explicit-id + restores; an unsorted supplied-id set reserves once per table before local + allocation; an id inside another session's cached range refuses. Cover + edge handles, keyed-edge reminting, Merge-by-id unkeyed endpoint moves, + rename rendering, lost import grants and independently assigned type ids. + - `integration/traversal_indexed.rs`, `integration/traversal_adaptive.rs`: + results unchanged on the existing fixtures with integer interning. + - `schema_apply.rs` and `integration/recovery.rs`: the upgrade's guard, + frozen source pins and ranges; crash after node conversion but before + edge conversion; unkeyed rows with identical properties; retained source + and staged pins during collection; crash before and after activation. + Verify historical read/export and the change-feed format-boundary refusal; + old-format branch creation and cross-format merge must refuse. + - `forbidden_apis.rs`: upgraded-format execution accepts `NodeRef`, never a + bare string node identity; the separate legacy history decoder remains + available for old-format reads. +- **GQT:** cases mirroring the merge truth table for the insert-insert case; + `@id` rendering and filters for unkeyed nodes and all edge kinds; endpoint + pseudo-properties; a key change by `update`; rename changes handle spelling + while preserving internal references. +- **DST:** concurrent branch inserts with merges, oracle "one row per key per + type on every branch and after every merge, every endpoint resolves". +- **Reference engine:** refuses the new IR. + +Acceptance thresholds: every existing GQT case passes unchanged except those +whose expectation prints a changed node/edge handle or relies on the former +key-immutability rule. A fully indexed single-property key batch uses one +probe per type; absent coverage scans with identical results. A +`identity_csr_hub` fixture with one hub and 10,000 keyed leaves measures zero +string interning in the CSR build. These are implementation obligations, not +measurements of this draft. + +Surfaces surveyed: `physical_table_schema`, `SystemFieldRole`, +`canonical_key_id`, the ULID mint in the loader, `TypeIndex`, the merge row +outcome classification in `exec/merge.rs`, `ChangeEndpointsOutput`, +`system_column_upgrade.rs`, Lance 11 scalar indexes on `UInt64` as probed by +the polymorphic RFC. + +## Rollout + +1. **Columns and sequence.** `__id: UInt64` on node and edge tables, packed + endpoints, `NodeRef`, key resolution on writes, the upgrade. Every existing + suite is adapted to the new identity contract. This is an internal build + stage; activation waits for merge, export and traversal support below. +2. **Merge and export.** The insert-insert rule, export and load rules, the + `__id` field and supplied-number reservation on load. +3. **Traversal on integers.** `TypeIndex` and the CSR build over packed + references. The format ships only when all three stages pass, including + retained-history compatibility and the upgrade recovery matrix. + +`implementation` moves to `in-progress` with phase 1 and to `complete` with +phase 3. + +## Unresolved questions + +- **Exposing keyed `__id`.** The numeric field is admitted only for unkeyed + rows in this version. Azim decides any future keyed-row preservation + contract, including natural-key conflicts, before admitting it. +- **Global ordinal space for the graph index.** Deferred to the polymorphic + RFC's later phase, whose authors own that decision; this RFC keeps per-type + dense spaces. + +## Decision log + +2026-10-12: use engine-minted row ids and packed 16/48 endpoint references; +keys remain public addresses. Accept the format and handle changes. Preserve +mutable-key identity across repeated merges through correspondence in accepted +history. The offline upgrade freezes source pins and numeric assignments +before conversion, and export/load distinguishes existing-row updates from +fresh supplied-id reservations. diff --git a/docs/rfcs/README.md b/docs/rfcs/README.md index 402e51a25..80afd788f 100644 --- a/docs/rfcs/README.md +++ b/docs/rfcs/README.md @@ -243,7 +243,10 @@ then dated RFCs by date. | [2026-09-30](2026-09-30-owned-server-operations.md) | Owned server operations | maintainer | accepted | complete | | [2026-09-30](2026-09-30-schema-contract-in-manifest.md) | Schema contract in the manifest | maintainer | draft | in-progress | | [2026-10-01](2026-10-01-engine-settlement-and-resource-bounds.md) | Engine settlement and resource bounds | maintainer | accepted | partial | +| [2026-10-07](2026-10-07-polymorphic-types.md) | Polymorphic types: interfaces, unions and polymorphic edges | maintainer | draft | not-started | | [2026-10-08](2026-10-08-index-facts-as-planner-input.md) | Index facts as planner input | maintainer | accepted | complete | | [2026-10-10](2026-10-10-unpublished-graph-commits.md) | Unpublished graph commits | maintainer | accepted | partial | | [2026-10-10](2026-10-10-merge-base-block-registry.md) | Block registry for merge base lookup | maintainer | draft | in-progress | | [2026-10-11](2026-10-11-unattached-graph-branches.md) | Unattached graph branches | maintainer | draft | not-started | +| [2026-10-12](2026-10-12-distributed-sequence.md) | Distributed sequence | maintainer | draft | not-started | +| [2026-10-12](2026-10-12-integer-node-identity.md) | Integer node identity | maintainer | draft | not-started | From 498ac7ce212678793c880274912a45784a446618 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 12 Oct 2026 15:13:58 -0700 Subject: [PATCH 2/4] rfc: add interface-bound UI consumer contracts on the identity baseline Contracts from a review against UI components bound to interfaces, which packed references do not cover: - Result schemas come from declared types and never depend on membership; a redeclaration may not drop an inherited @key or @unique. - `@type_id` is an expression separate from `@type`: the identity source is `~node_type_id`, the arm's StableTypeId or the packed destination's type half; `~node_type` is derived from it through the captured catalog. - The union scan resolves filter, index access, statistics and late materialization per arm; on main every single-table shortcut broke. - Pruning runs after parameter binding, for reads and interface-wide mutations; a non-member type is refused. - The catalog route's field list and schema hash, per-commit feed expansion with a membership refusal, refusal of policy-denied members, interface-property migration rows, one namespace with stored-query pinning, and concurrent arms with a handle cache sized to the catalog. --- docs/rfcs/2026-10-07-polymorphic-types.md | 329 ++++++++++++++++++---- 1 file changed, 277 insertions(+), 52 deletions(-) diff --git a/docs/rfcs/2026-10-07-polymorphic-types.md b/docs/rfcs/2026-10-07-polymorphic-types.md index e7c2aef4a..065051e04 100644 --- a/docs/rfcs/2026-10-07-polymorphic-types.md +++ b/docs/rfcs/2026-10-07-polymorphic-types.md @@ -132,8 +132,12 @@ edge RelatedTo: Subject -> Subject constraint forms remain refused until their enforcement is specified. Property-level annotations on interface properties keep today's meaning: they are applied to each implementor that inherits the property. - Redeclaring an interface property in a node and dropping one of its - constraint annotations is a lint warning, no longer a silent drop. + A node that redeclares an interface property must give it the identical + type, as today, and may add constraints but never drop an inherited `@key` + or `@unique`. Today a redeclared `slug: String` silently loses the + interface's `@key`: in the prototype the same slug then loaded as two rows. + With keys as the public address, that silently breaks addressing, so it + becomes a refusal. Dropping any other inherited annotation is a lint warning. - An edge endpoint is a node type, an interface, or an inline union `A | B | …` whose members are node types or interfaces. Its member set is the union of each member's concrete closure. @@ -189,12 +193,29 @@ query non_people($q: String) { |---|---| | `$x: I` | Every concrete implementor of interface `I`, through interface inheritance. | | `$x: A \| B` | The union of the members' sets; members may be interfaces. | -| `$x.p` | Legal when `p` is declared on `I` (or inherited by it). For a union, legal when every member has `p` under the alternation rule: same scalar and list shape, nullable if any member is, enum domains combined, enum with String becomes String. Otherwise a type error that names the narrowing to use. | -| `$x.@type` | The concrete type's canonical name, a non-null String. | +| `$x.p` | Legal when `p` is declared on `I` (or inherited by it), and typed exactly as `I` declares it. For a union, each declared member contributes one type (a node type its property type, an interface its declared type) and the alternation rule joins those: same scalar and list shape, nullable if any member is, enum domains combined, enum with String becomes String. Otherwise a type error that names the narrowing to use. | +| `$x.@type` | The concrete type's canonical name, a non-null String. A rename changes it. | +| `$x.@type_id` | The concrete type's `StableTypeId`, the type half of the node's packed reference, as a non-null decimal String. A rename keeps it, so clients cache and route by it. | | `$x is T`, `$x is not T`, `$x is A \| B` | A non-null Bool type test. `T` must overlap the binding's set. | | `$x: T` after an earlier binding of `$x` | Intersects the two sets. An empty intersection is a type error. Rebinding to a subtype is no longer T50. | | `$e.@src_type`, `$e.@dst_type` | The endpoint types of a bound edge, as canonical names, read from the packed references. | -| `return { $x }` | Unchanged for a concrete binding. For a multi-member set, a struct of `@id`, `@type` and the readable properties. | +| `return { $x }` | Unchanged for a concrete binding. For an interface or union binding, a struct of `@id`, `@type` and the readable properties, even while the set has one member. | + +***Result schemas never depend on the member set.*** Adding or removing an +implementor changes which rows an abstract binding returns, never a column's +type, the legality of a property read, or the shape of `return { $x }`. A UI +component that declares the row shape a stored query must fill can rely on +that. The prototype confirmed it: applying a new implementor left a calendar +query's result schema byte-identical. Joining a union over its concrete +closure instead would break this: the first implementor added to an interface +inside a union could turn a column nullable or widen an enum to `String`. + +Ordering ties break on the packed reference, so an abstract binding's total +order is its `order` keys, then `@type_id`, then the row id. That makes keyset +paging by filters exact across members, because each arm evaluates the filter +on its own table. A paging surface for stored queries (a cursor carrying the +snapshot and the last row's order tuple) is a separate RFC; it needs this +total order and a `limit` that can be a parameter, which GQ lacks today. Traversal endpoints use the same sets. A traversal's destination set is the edge's endpoint set, intersected with any declared type of the destination @@ -217,9 +238,12 @@ recurses across Person and Organization with one shortest-distance search. Refusals: -- `bm25`, `search`, `fuzzy` and `match_text` on a binding whose set has more - than one member are refused. Full-text scores are computed per dataset and - are not comparable across tables. +- `bm25` on a binding whose set has more than one member is refused. Lance + computes BM25 statistics per index, so scores are not comparable across + tables. +- `search`, `fuzzy` and `match_text` produce no score; they are membership + filters. They run per arm when every member has an inverted index on the + property, and are refused, naming the member, otherwise. - `nearest` is admitted when the vector property and its `@embed` model are declared on the interface and every member records the same model and dimension. @@ -255,8 +279,16 @@ query touch($slug: String) { - `update I set … where …` and `delete I where …` are typed against the interface. They apply to every member and publish once, under the existing rule that a query cannot mix writes with deletes. Assignments may name only - properties readable on `I`; the `where` may use those properties, `@id` and - `@type`. + properties readable on `I`; the `where` may use those properties, `@id`, + `@type` and `@type_id`. +- Arms are resolved after parameters are bound, so pruning applies to + parameterised type predicates too. A component that edits a row by its type + and address (`where @type_id = $t and slug = $s`) touches one table instead + of opening every member. A bound type that is not a member of `I` is a typed + refusal, not zero rows. A member whose predicate matches nothing stages + nothing (on main a zero-match update already stages and publishes nothing), + so an N-member update costs one staged table commit per matching member and + one manifest publication. - A polymorphic endpoint with more than one member is written with its concrete type: `Type($id)` in GQ, `from_type` or `to_type` in a load or graph-batch envelope. The type must be a member of the endpoint set. @@ -275,7 +307,10 @@ query touch($slug: String) { - `omnigraph schema plan` reports the new step kinds (below) and their tier. - An abstract binding opens one dataset per member. A cold first read pays one dataset open per member type, and arms open concurrently within the query's - resource bounds. + resource bounds. The table-handle cache holds at least every table in the + accepted catalog. Today it holds 64, so a 75-member interface evicts its own + handles: in the prototype a warm repeat of a 75-member keyed lookup reopened + 55 to 61 tables, against none at 60 members. - Every schema migration in this RFC is a schema apply, and schema apply refuses while any branch other than `main` exists. A graph that keeps long-lived branches lands them before widening an endpoint. @@ -363,9 +398,22 @@ upgraded current format, and historical abstract bindings remain refused. forward, reverse, two-hop and negation traversals with one scan per edge table and rows routed by their endpoint types, and no per-pair members were needed. -- `$x.@type` lowers to a String literal for a singleton set and to a - query-only `~node_type` column otherwise, as `$e.@type` does with - `~edge_type`. A type test remains a typed, non-null Boolean expression. +- `$x.@type_id` and `$x.@type` are two expressions over one identity source, + the concrete type of the row. For a singleton set both fold to literals. + Otherwise the identity source is the query-only `~node_type_id` column + (`UInt64`): each union-scan arm emits its member's `StableTypeId` as a + constant, and an expansion emits the type half of the packed destination. + `~node_type` (Utf8) is derived from it through the captured catalog's + canonical names, never stored or carried as identity. Dedup, cycle closing, + tie-breaks and cursors use the packed reference, whose high half is + `~node_type_id`; only `@type` projections and comparisons read the name. + `@type_id` is projected as a decimal string, because a `u64` is unsafe as a + JSON number past 2^53. Literals and parameters compared with it parse to + `u64`, and a non-numeric bound value is a typed refusal. Neither column is + pushed into a member scan: before predicates are placed, each arm folds both + to its constants. `filter_pushable` refuses query-only columns. In the + prototype, `$e.@type = "Task"` was pushed into the member scans and failed + with "Unable to get field named `~node_type`". A type test remains a typed, non-null Boolean expression. Within each arm, its concrete type can replace the test with a Boolean constant and simplify the enclosing expression. Projecting a type test never removes rows; negation and disjunction preserve Boolean semantics. @@ -390,28 +438,55 @@ upgraded current format, and historical abstract bindings remain refused. ### Planning and execution - **Union scan.** An abstract binding becomes one new logical and physical - node over N pinned table specs, one per member. It lowers through `Lower` to - one OmniGraph operator, so a plan keeps one operator per node. Each arm - carries its own pushed filter and projection, resolved through the member's - satisfying property. Each arm is cast to the binding's joined types and - gains a Utf8 `~node_type` column; the plan mirrors carry no Arrow - `Dictionary` or `Union` type. The operator streams arms and opens their - datasets concurrently. DataFusion's `UnionExec` is not used unchanged: its - output partitioning is the sum of its inputs', the engine executes only - partition 0 of the root, and it never compares input types. + node over N arms, one per member. It lowers through `Lower` to one OmniGraph + operator, so a plan keeps one operator per node. Each arm is resolved on its + own: + - its table pin and projection; + - its filter, resolved through the member's satisfying property, with the + type columns folded to the arm's constants; + - its index facts, scan-access decision and statistics; + - its late materialization, where refetched rows carry their arm and go + back to that member's table. + + Each arm is cast to the interface's declared types and gains the + `~node_type_id` and `~node_type` columns; the plan mirrors carry no Arrow + `Dictionary` or `Union` type. The operator opens and streams arms + concurrently, within the query's I/O bounds, and keeps member order. + DataFusion's `UnionExec` is not used unchanged: its output partitioning is + the sum of its inputs', the engine executes only partition 0 of the root, + and it never compares input types. +- **No single-table shortcut sees an abstract type.** The prototype first + intercepted interfaces inside one scan function. Merged with main + (`80b3cfeb`), every path that keys on one table broke or degraded: + - the pipelined table read opened `node:`; + - late materialization (`HydrateColumns`) refused every ordered, limited + interface query, because it refetches by row address from one table; + - scan access recorded `sequential` for the interface, which replays as no + scalar index on every arm, while the same filter on one member used an + `index_probe`; + - an unfiltered `count($x)` became one metadata count on a table that does + not exist. + + Each of these consumers takes the union node and resolves per arm, or + refuses it. A test asserts that no table key of an abstract type reaches + `open_lance_dataset`. - **Pruning.** A rebinding intersection removes excluded arms. A filtering expression containing type tests or `@type` comparisons removes an arm only when simplifying the complete filter proves that the arm cannot contribute - a row. Pruned members are never opened. -- **Cost.** Estimates sum across members. Key equality bounds each member to - one row. `count($x)` over an unfiltered abstract binding sums one - `MetadataCount` per member. + a row. Parameters are bound by then, so `@type = $type` and + `@type_id = $t` prune too. Pruned members are never opened. +- **Cost.** Estimates and statistics are per arm and sum across arms. Key + equality bounds each member to one row through its own index. `count($x)` + over an unfiltered abstract binding sums one `MetadataCount` per member. + Explain shows each arm's access path. - **Expansion.** A polymorphic expansion is a selection, so it declares `ExpandPolicy::Budgeted` and `IndexedScan` and shares the traversal-work counter. One `src IN (frontier)` scan of an edge table serves every destination type; rows are routed by the type half of their packed destination. Admission charges each edge table's physical rows once per - probe window. `ExpandExec` emits `{dst}.~node_type` beside `{dst}.__id`. The + probe window. `ExpandExec` emits `{dst}.~node_type_id` (the packed + destination's type half) and the derived `{dst}.~node_type` beside + `{dst}.__id`. The interner, frontier and visited set are keyed by the packed reference, which makes mixed-type recursion correct where colliding keys would otherwise merge distinct nodes. @@ -496,27 +571,71 @@ upgraded current format, and historical abstract bindings remain refused. | Rename interface (`@rename_from`) | supported | none | | Add `implements` when the node already declares every inherited property compatibly | supported; validated if a uniqueness obligation is introduced or widened | none; satisfaction links change | | Add `implements` with missing nullable properties | supported; validated if a uniqueness obligation is introduced or widened | today's add-property path for those properties | +| Add a node type that implements an interface | supported | a new table; no result schema changes | +| Add a nullable interface property | supported (already on main) | metadata-only `AddProperty` on every inheriting member, in one publication; a member that already declares it identically gains a link | +| Add a required interface property | refused (OG-MF-103) | metadata only when every member already declares it identically, or there are no members | +| Rename an interface property (`@rename_from`) | supported (already on main) | `RenameProperty` on each member; a member that redeclares it needs its own matching `@rename_from` in the same migration | +| Widen an interface property's enum | supported (already on main) | `ExtendEnum` on each member | +| Remove an interface property | destructive, per member, each named in the plan | the column leaves every inheriting member's table; members that redeclare it keep theirs; refused while a stored query reads it through the interface | +| Other interface property type or nullability change | refused (OG-MF-106) until phase 4 | | | Generalize an endpoint from `C` to an interface or union containing `C` | supported, keyed and unkeyed edges alike | none; existing rows already carry `C` in their packed endpoints | | Remove `implements`, remove an interface, narrow an endpoint | validated (the OG-MF-104 tier) | refused while any row's endpoint names a removed member, or a registered stored query depends on it | +| Reuse a name released by a rename or drop, as a different kind, in the same apply | refused | "interface 'Event' reuses the name node 'Event' gave up in this migration; apply the rename first" | | Fold an edge family into one polymorphic edge | explicit copy | rows of the named source edge types are copied into the new edge; the sources are tombstoned | Schema apply keeps its main-only, single-live-branch restriction and its one manifest publication. Every step above fits that publication. +A stored query ***depends on*** member `M` of `I` when it fails to typecheck +against the desired catalog: type tests, rebindings and `@type`/`@type_id` +literals naming a non-member are type errors. A parameterised type has no +static dependency and is refused at run time. Mentioning `I` alone is not a +dependency, so one view over an interface never blocks removing `implements` +from a member. The data dependency is any row whose endpoint names `M`. +Cluster apply preflight already typechecks the desired queries against the +desired catalog, so a query and the schema can change in one apply; a +standalone apply has no registry, so served startup refuses instead. The +registry also records, per stored query, each binding name's kind and +`StableTypeId` and the result descriptor, and refuses when a name later +resolves to a different kind or id or the descriptor changes. A fork that +renames `node Event` and later adds `interface Event` therefore cannot silently +widen an old `$e: Event` view. + ### Wire, API and descriptor -- Read results gain a `@type` column only where a query projects it. Abstract - bindings encode as ordinary flat columns. Arrow-json has no `Union` encoder, - and the descriptor refuses `Union`, so no result uses one. +- Read results gain `@type` and `@type_id` columns only where a query projects + them. Abstract bindings encode as ordinary flat columns. Arrow-json has no + `Union` encoder, and the descriptor refuses `Union`, so no result uses one. +- Every read response carries the schema hash (`schema_ir_hash`) as a field and + a header, so a client knows which catalog a type name belongs to. Today a read + carries only `graph_commit_id`, and no route exposes a type id or schema hash. - `ChangeEndpointsOutput`, export lines and load envelopes gain optional `from_type`/`to_type`, present only for polymorphic endpoints. -- A structured, read-gated catalog route lists node types, interfaces with - their members, and edges with expanded endpoint sets. Clients stop - re-deriving implementor sets from `.pg` text. OpenAPI parity tests force - each consumer to classify it. -- Type-name filters on the change feed and export accept an interface name. - They expand it to members when the cursor is created and record the - expansion in the cursor. +- A structured, read-gated catalog route returns what a generic shell needs to + show views and build create and edit forms. Today only `/schema` returns + schema information, as raw `.pg` text. The route returns: + - per type: `type_id` (decimal string), kind, implemented interfaces + (closure), and type-level `@description`/`@instruction`; + - per property: name, scalar kind, list, vector dimension, nullability, enum + values, annotations, and whether a form must not ask for it (`@embed` or + generated); + - `@key` and `@unique` tuples; + - per interface: properties, inherited interfaces and current members; + - per edge: cardinality, properties and expanded endpoint sets; + - top level: `schema_identity_domain` and the schema hash, also sent as an + `ETag`. + + Clients stop re-deriving implementor sets from `.pg` text. OpenAPI parity + tests force each consumer to classify it. +- Type-name filters on the change feed and export accept an interface name. It + is kept as written and expanded per commit, against that commit's accepted + catalog, as main already matches type names per commit interval. When a + polled commit changes the member set of a filtered interface, the poll ends + with a typed `ChangeScopeMembershipChanged` refusal carrying the added and + removed members and a cursor positioned before that commit; the client + re-baselines. Adding `implements` writes no rows, so a feed cannot report + the new member's existing rows as changes. Expanding once at cursor creation + would freeze membership, and a live view would miss a member added later. - The operation descriptor lists every member type in `reads` and `writes`. ### Policy @@ -527,8 +646,21 @@ read of the same graph on the same branch, and an interface-wide mutation is a `change`, so nothing changes. When per-type rules arrive (the planned `predicate_for`), the planner expands each abstract binding to concrete types before evaluating policy. `implements` never confers a permission, because a -`schema_apply` holder could otherwise widen read access. A traversal treats an -endpoint whose type is denied as filtered, never as a dangling reference. +`schema_apply` holder could otherwise widen read access. + +A denied member never silently thins a result: +- A read whose binding or traversal destination includes a member the actor + may not read is refused with a typed `policy_denied_member` error, unless the + query removes that member (a type test, a rebinding, or type pruning). +- An interface-wide mutation with a denied arm is refused before staging, + never partly applied. +- Row-level predicates inside an allowed type may filter. The response then + carries `policy_filtered: true`, with no counts, so a view can say its + results are partial. + +Filtering a denied arm instead would make `count` and a view's completeness +wrong with no signal. A denied endpoint is never rendered as a dangling +reference. ## Invariants @@ -547,8 +679,9 @@ endpoint whose type is denied as filtered, never as a dangling reference. interface projection change cost, never membership or results. - **8, loud integrity failures.** Ambiguous direction, an unknown or non-member endpoint type, an untyped polymorphic endpoint, cross-member - cardinality violations and an untyped expansion over a polymorphic edge are - typed refusals. + cardinality violations, an untyped expansion over a polymorphic edge, a + non-member type in an interface-wide mutation, a policy-denied member and a + feed membership change are typed refusals. No result is silently partial. - **9, typed semantics.** Type sets, selections, type tests and pruning live in AST, IR and plan structures. Type filters are structured predicates. - **11, bounded resources.** Union arms run within the query's memory and @@ -592,6 +725,10 @@ sub-pipelines), no cost-blind plan choice, and no maintained parallel truth. | Concrete supertypes (`node Employee extends Person`) | Raises key-scope and reclassification questions. Knowledge graphs model this by composition. Deferred. | | Named unions as a declaration kind | Marker interfaces cover reuse. Inline unions cover closed endpoint sets. Deferred. | | Resolve untyped polymorphic endpoints by probing every member table | Hides an N-table cost in every write, and turns colliding keys into errors at write time. Typed endpoints are the default, and probing is an unresolved question. | +| Join a union's property types over its concrete closure | Matches the alternation rule literally, but adding an implementor to an interface inside a union could change a stored query's result schema. Joining declared member types keeps result schemas independent of membership. | +| Filter a policy-denied member's rows | Simple, but makes `count` and a view's completeness silently wrong. Refusing unless the query removes the member keeps results whole. | +| Expand an interface feed filter once, when the cursor is created | Needs no new refusal, but freezes membership: a live view misses every member added later. | +| Normalize BM25 scores across arms, or fuse arms with `rrf` | Normalization is unstable across `k`, filters and pages. Rank fusion over N arms keyed by packed reference is plausible later; today's `rrf` takes exactly two arms and fuses by bare id, which would merge colliding keys. | Prior art. Gel (EdgeDB) keeps a table per concrete type and compiles abstract reads to a per-query `UNION ALL`, with `__type__` as a per-table constant. @@ -670,6 +807,48 @@ What the prototype found, and where each landed: With the prototype's changes, the existing suites still passed: 465 compiler tests, the planner suites, 706 engine tests, and all 275 GQT cases. +The prototype was then merged with main at `80b3cfeb` (now at `00bee952`, +findings also in `UI_BINDING_FINDINGS.md`). On the merged build: 471 compiler +tests, the planner suites (115), 468 engine library tests, 573 engine +integration tests (with CI's `RUST_MIN_STACK`), and all 333 GQT cases pass, +two after a rerun at a 10-second timeout under machine load. The merged build +still uses string identity; its union-scan findings below do not depend on the +identity representation. + +### UI components bound to interfaces + +A second review tested the design against a consumer it was not drawn around: +forkable templates whose components (calendar, kanban, timeline, inbox) bind to +stored queries over interfaces, read the catalog route, edit through +interface-wide `update`, and identify rows by type and address. Twelve probes +ran against the prototype, before and after the merge with main; claims about +code were spot-checked on `origin/main`. + +| # | Question | Finding | Where it lives now | +|---|---|---|---| +| 1 | Result typing | Members must match the interface type exactly, so interface columns are stable; union joins over the concrete closure would not be. A redeclared key property silently loses `@key`. | Declared-type rule under Queries; inherited-key refusal under Schema. | +| 2 | Interface-property migrations | Main already fans out add, drop, rename and enum widening to every member; the table covered none. | Migration table. | +| 3 | Stable type identity | Renames keep `StableTypeId`, but nothing exposed it or a schema hash. | `@type_id`, schema hash, catalog route. | +| 4 | Order, limit, paging | Correct while the union is a leaf; on main every ordered, limited interface query failed in `HydrateColumns`. No paging surface. | Per-arm late materialization; packed-reference tie-break; paging deferred. | +| 5 | Type filters and counts | `@type = "Task"` was pushed into member scans and failed; a bare `count` failed. | Per-arm folding; pruning after parameter binding; member-wise counts. | +| 6 | Indexes per arm | On main the plan records `sequential` for every arm. | Per-arm access, shown in explain. | +| 7 | Search | `search`/`fuzzy`/`match_text` are membership filters, refused needlessly. | Refusal narrowed to `bm25`. | +| 8 | Wide interfaces | 75 members: about 10× a single-type lookup locally; the 64-entry handle cache reopened 55 to 61 tables per warm query. | Concurrent arms; cache sized to the catalog; threshold below. | +| 9 | Component writes | Zero-match members stage nothing, but every member was opened. | Arm pruning after parameter binding. | +| 10 | Catalog route | Too thin for a create form; no schema version. | Field list and `ETag`. | +| 11 | Live views | Expanding at cursor creation freezes membership. | Per-commit expansion and membership refusal. | +| 12 | Remove-`implements` dependency | "Depends" was undefined. | Typecheck-based rule and registry pinning. | +| 13 | Per-type policy | A denied member was silently filtered. | Refusal unless narrowed. | +| 14 | Shared core name collision | `interface Event` and `node Event` coexist on main; reusing a renamed name changes stored queries silently. | One namespace; same-apply reuse refused; registry pinning. | + +Packed references dissolve none of these: they concern typing, planning, +catalog and lifecycle contracts, not stored identity. Local measurements for +the wide case (75 members, local disk): an Entity-wide keyed lookup took 38 to +47 ms cold and 28 to 37 ms warm with arms run one after another, and 25 to +28 ms cold and 20 to 23 ms warm with 16 concurrent arms, against about 3 ms cold +and 2 ms warm for one type. The S3 figure in the acceptance threshold is a +target, not a measurement. + ### Tests to extend Following [the test map](../dev/testing.md), these existing owners are @@ -699,15 +878,29 @@ extended: A pre-phase-3 binary refuses `interface-unique`, including empty interfaces. - Generalizing a keyed edge keeps every existing row matchable by its key. - Plan-replay tests: every member pin. + - No table key of an abstract type reaches `open_lance_dataset`, across the + pipelined read, late materialization, scan access, metadata count and + contains join. + - An ordered, limited interface query returning vector and struct columns. + - Adding an implementor leaves every stored query's result descriptor + unchanged; redeclaring an inherited `@key` without it is refused. + - Type filters prune arms after parameter binding; a non-member type in an + interface-wide mutation is refused; zero-match arms stage nothing. - **GQT:** cases that mirror the issue-659 suite. They cover union scans, narrowing, projected type-test true/false values without row loss, negated tests and disjunctions accepted by the other operand, `@type` ordering with colliding keys, mixed-type recursion, empty interfaces, historical refusal, `bm25` refusal and forced-CSR refusal before phase 3, on the local filesystem and on - deterministic object storage. + deterministic object storage. One case projects and filters both `@type` + and `@type_id`, renames a member, and checks that the name changes while the + stable id does not, both while the interface has one member and after it + grows to two. - **DST:** concurrent writes to a polymorphic edge with cascades. - **Reference engine:** refuses the new IR. +- **Server:** catalog-route fields and `ETag`; schema hash on reads; feed + `ChangeScopeMembershipChanged`; `policy_denied_member` once per-type rules + exist. - **Tower:** catalog-route parity and grammar capability gating. Acceptance thresholds: @@ -720,6 +913,8 @@ Acceptance thresholds: refusal, as the alternation suite does. - Generalizing an endpoint on a table of one million edges touches no data file. +- A keyed lookup on a 75-member interface on S3 completes within 2× a single + keyed lookup when warm, and opens no table twice. ### Defects found during the investigation @@ -730,6 +925,11 @@ Acceptance thresholds: - Tower's insert-edge key pre-check finds the edge by `(name, from, to)`. For a polymorphic edge it would skip the check, and a keyed insert would upsert. Tower changes this before it enables polymorphic edges. +- A node that redeclares an inherited key property silently loses the + interface's `@key` today, so the same slug loads as two rows. Phase 0 + refuses it. +- `interface Event` and `node Event` can coexist today; the node wins name + resolution. ## Rollout @@ -739,22 +939,28 @@ graph. 1. **Phase 0, groundwork.** - Validate interface properties at declaration. - - Warn when a redeclaration drops an interface constraint. - - Refuse new interface and node name collisions; support interface rename. + - Refuse a redeclaration that drops an inherited `@key` or `@unique`; warn + on other dropped annotations. + - Refuse new interface and node name collisions and same-apply name reuse + across kinds; support interface rename. - Allow metadata-only `implements` additions, subject to schema validation once interface-wide constraints exist. - Make the three lowering sites handle type sets. 2. **Phase 1, abstract bindings.** - - Query surface: interface and union bindings, `@type`, type tests and - narrowing. - - Execution: union scans, interface-wide `nearest`, and interface-wide - `update`/`delete`. - - Interface inheritance behind `interface-inheritance`, and the catalog - route. + - Query surface: interface and union bindings, `@type`, `@type_id`, type + tests and narrowing. + - Execution: the union-scan node with per-arm resolution, covering every + single-table consumer; pruning after parameter binding; concurrent arms + and a handle cache sized to the catalog; interface-wide `nearest`, and + interface-wide `update`/`delete`. + - Interface inheritance behind `interface-inheritance`; the catalog route + with its full field list; the schema hash on reads; stored-query pinning + of binding kinds and ids. 3. **Phase 2, polymorphic edges.** - Endpoint declarations behind `polymorphic-endpoints`. - Typed endpoint writes, validation, cascade, loads, export and - change-feed endpoint types. + change-feed endpoint types; per-commit expansion of interface filters with + the membership refusal. - Budgeted indexed traversal. - Generalization changes schema metadata only. Folding stages copied rows and source-table retirement, then publishes them together. @@ -794,6 +1000,14 @@ required to admit phases 1 and 2. - **Interface-named policy rules.** Should they exist under per-type authorization, compiled to the current member set and re-checked on every schema apply? +- **Paging surface.** An opaque cursor on stored-query invocation, carrying the + snapshot and the last row's order tuple, or GQ syntax such as + `after $cursor`? Either needs the total order and a `limit` that can be a + parameter. +- **Feed membership changes.** Refuse with `ChangeScopeMembershipChanged`, as + specified, or report membership events inline in the feed? +- **Removing an interface property.** One step that drops a column from N + tables, or a plan that must list each member explicitly? ## Decision log @@ -814,3 +1028,14 @@ endpoint generalization is metadata-only even for keyed edges. Folding still copies rows. Type tests retain Boolean expression semantics. Interface-wide uniqueness ships with its own feature fence and schema-admission validation; export/load preserves the identity RFC's logical-key contract across graphs. + +2026-10-12: added the contracts from a review against interface-bound UI +components, which packed references do not cover: declared-type result +schemas, `@type_id` as an expression separate from `@type` over the packed +type half, a union-scan node resolved per arm (every single-table shortcut +broke on main), pruning after parameter binding for reads and interface-wide +mutations, the catalog route's fields and schema hash, per-commit feed +expansion with a membership refusal, refusal of policy-denied members, +interface-property migration rows, the inherited-key refusal, one namespace +with stored-query pinning, and concurrent arms with a handle cache sized to the +catalog. From a078bf047bb2653e20941f1f61ec8d9d9b7906c1 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 12 Oct 2026 18:10:28 -0700 Subject: [PATCH 3/4] rfc: defer polymorphic-type visibility to actor views Abstract type sets resolve against the actor's schema view, and view.restricted replaces the policy_denied_member refusal and the policy_filtered marker: the refusal disclosed that a hidden member exists, and its only escape required naming a type the caller may not know. An inherited placement cannot be dropped or added per member, and the catalog route, schema hash, descriptors and feed refusals describe the schema view. Resolves the interface-named policy rules question. --- docs/rfcs/2026-10-07-polymorphic-types.md | 89 ++++++++++++++++------- 1 file changed, 61 insertions(+), 28 deletions(-) diff --git a/docs/rfcs/2026-10-07-polymorphic-types.md b/docs/rfcs/2026-10-07-polymorphic-types.md index 065051e04..0f1231cc0 100644 --- a/docs/rfcs/2026-10-07-polymorphic-types.md +++ b/docs/rfcs/2026-10-07-polymorphic-types.md @@ -137,7 +137,11 @@ edge RelatedTo: Subject -> Subject or `@unique`. Today a redeclared `slug: String` silently loses the interface's `@key`: in the prototype the same slug then loaded as two rows. With keys as the public address, that silently breaks addressing, so it - becomes a refusal. Dropping any other inherited annotation is a lint warning. + becomes a refusal. A redeclaration that drops an inherited placement (`@in`, + from [Actor views and subgraph sharing](https://github.com/ModernRelay/omnigraph/pull/951)) is refused for the same reason, since it would expose the + column; a member also may not add a fixed placement to an inherited + property, so the property's visibility is the same for every member. + Dropping any other inherited annotation is a lint warning. - An edge endpoint is a node type, an interface, or an inline union `A | B | …` whose members are node types or interfaces. Its member set is the union of each member's concrete closure. @@ -637,30 +641,52 @@ widen an old `$e: Event` view. the new member's existing rows as changes. Expanding once at cursor creation would freeze membership, and a live view would miss a member added later. - The operation descriptor lists every member type in `reads` and `writes`. +- Under an actor view (see [Policy](#policy)) each of these surfaces describes + the actor's schema view: the catalog route lists only visible types, + properties and members; reads carry the schema view's digest in place of + `schema_ir_hash`, so a change to a hidden type is not observable; + operation descriptors list visible members; and a + `ChangeScopeMembershipChanged` refusal names only visible members. ### Policy Cedar has no type dimension today: its resources are graph, server and cluster, and a request is action, branch and actor. An abstract binding is a read of the same graph on the same branch, and an interface-wide mutation is a -`change`, so nothing changes. When per-type rules arrive (the planned -`predicate_for`), the planner expands each abstract binding to concrete types -before evaluating policy. `implements` never confers a permission, because a -`schema_apply` holder could otherwise widen read access. - -A denied member never silently thins a result: -- A read whose binding or traversal destination includes a member the actor - may not read is refused with a typed `policy_denied_member` error, unless the - query removes that member (a type test, a rebinding, or type pruning). -- An interface-wide mutation with a denied arm is refused before staging, - never partly applied. -- Row-level predicates inside an allowed type may filter. The response then - carries `policy_filtered: true`, with no counts, so a view can say its - results are partial. - -Filtering a denied arm instead would make `count` and a view's completeness -wrong with no signal. A denied endpoint is never rendered as a dangling -reference. +`change`, so nothing changes at that gate. + +Per-type and per-row visibility is defined by +[Actor views and subgraph sharing](https://github.com/ModernRelay/omnigraph/pull/951), and abstract types follow it: + +- An abstract binding resolves its type set against the actor's schema view. + A member type the actor cannot see is absent from the set, exactly as it + would be absent from a graph holding only the actor's view. Visibility is + decided per concrete type; `implements` neither grants nor removes it, so a + `schema_apply` holder cannot widen read access by adding `implements`. +- A type test, or a `@type` or `@type_id` comparison, naming a type hidden + from the actor fails with the unknown-name error a misspelling produces. +- Each union arm whose member the view restricts carries that member's row + predicate, placed during per-arm resolution. A traversal never reaches a + node the actor cannot see, so a hidden endpoint is never rendered as a + dangling reference. +- An interface-wide mutation checks every arm against the actor's view before + staging and is never partly applied. +- When the view removed a member or filtered rows of any type the plan read, + the response's `view.restricted` is `true`. It is computed from the view and + the plan, never from data, and carries no counts. A view that must be + complete refuses on it. + +Result schemas still never depend on the member set, because they come from +declared types. + +An earlier draft refused a read whose type set included a denied member +(`policy_denied_member`) unless the query removed that member, and marked +row-level filtering with `policy_filtered`. Both are withdrawn. The refusal +disclosed that a hidden member exists, and its only escape, a type test that +removes the member, required the caller to name a type it is not allowed to +know. A stored query shared by actors with different views could never be +written that way. `view.restricted` gives a view the same completeness signal +without naming a member. ## Invariants @@ -680,8 +706,9 @@ reference. - **8, loud integrity failures.** Ambiguous direction, an unknown or non-member endpoint type, an untyped polymorphic endpoint, cross-member cardinality violations, an untyped expansion over a polymorphic edge, a - non-member type in an interface-wide mutation, a policy-denied member and a - feed membership change are typed refusals. No result is silently partial. + non-member type in an interface-wide mutation and a feed membership change + are typed refusals. A result filtered by an actor view says so in + `view.restricted`; no result is silently partial. - **9, typed semantics.** Type sets, selections, type tests and pruning live in AST, IR and plan structures. Type filters are structured predicates. - **11, bounded resources.** Union arms run within the query's memory and @@ -726,7 +753,7 @@ sub-pipelines), no cost-blind plan choice, and no maintained parallel truth. | Named unions as a declaration kind | Marker interfaces cover reuse. Inline unions cover closed endpoint sets. Deferred. | | Resolve untyped polymorphic endpoints by probing every member table | Hides an N-table cost in every write, and turns colliding keys into errors at write time. Typed endpoints are the default, and probing is an unresolved question. | | Join a union's property types over its concrete closure | Matches the alternation rule literally, but adding an implementor to an interface inside a union could change a stored query's result schema. Joining declared member types keeps result schemas independent of membership. | -| Filter a policy-denied member's rows | Simple, but makes `count` and a view's completeness silently wrong. Refusing unless the query removes the member keeps results whole. | +| Refuse a type set that includes a member hidden from the actor (`policy_denied_member`) | Keeps results whole, but discloses that the hidden member exists, and the only escape names the hidden type. Filtering over the actor's schema view with `view.restricted` gives the same completeness signal without disclosure. | | Expand an interface feed filter once, when the cursor is created | Needs no new refusal, but freezes membership: a live view misses every member added later. | | Normalize BM25 scores across arms, or fuse arms with `rrf` | Normalization is unstable across `k`, filters and pages. Rank fusion over N arms keyed by packed reference is plausible later; today's `rrf` takes exactly two arms and fuses by bare id, which would merge colliding keys. | @@ -838,7 +865,7 @@ code were spot-checked on `origin/main`. | 10 | Catalog route | Too thin for a create form; no schema version. | Field list and `ETag`. | | 11 | Live views | Expanding at cursor creation freezes membership. | Per-commit expansion and membership refusal. | | 12 | Remove-`implements` dependency | "Depends" was undefined. | Typecheck-based rule and registry pinning. | -| 13 | Per-type policy | A denied member was silently filtered. | Refusal unless narrowed. | +| 13 | Per-type policy | A denied member was silently filtered. | Member sets from the actor's schema view, with `view.restricted` as the completeness signal. | | 14 | Shared core name collision | `interface Event` and `node Event` coexist on main; reusing a renamed name changes stored queries silently. | One namespace; same-apply reuse refused; registry pinning. | Packed references dissolve none of these: they concern typing, planning, @@ -899,8 +926,9 @@ extended: - **DST:** concurrent writes to a polymorphic edge with cascades. - **Reference engine:** refuses the new IR. - **Server:** catalog-route fields and `ETag`; schema hash on reads; feed - `ChangeScopeMembershipChanged`; `policy_denied_member` once per-type rules - exist. + `ChangeScopeMembershipChanged`; once actor views exist, member sets from the + schema view, identical errors for hidden and undeclared type names, and + `view.restricted`. - **Tower:** catalog-route parity and grammar capability gating. Acceptance thresholds: @@ -997,9 +1025,6 @@ required to admit phases 1 and 2. - **Historical abstract bindings.** When can they be admitted? That needs historical membership, which the captured contract does not reconstruct today. -- **Interface-named policy rules.** Should they exist under per-type - authorization, compiled to the current member set and re-checked on every - schema apply? - **Paging surface.** An opaque cursor on stored-query invocation, carrying the snapshot and the last row's order tuple, or GQ syntax such as `after $cursor`? Either needs the total order and a `limit` that can be a @@ -1039,3 +1064,11 @@ expansion with a membership refusal, refusal of policy-denied members, interface-property migration rows, the inherited-key refusal, one namespace with stored-query pinning, and concurrent arms with a handle cache sized to the catalog. + +2026-10-12: adopted +[Actor views and subgraph sharing](https://github.com/ModernRelay/omnigraph/pull/951) +for per-type visibility. Abstract type sets resolve against the actor's schema view, and `view.restricted` replaces the +`policy_denied_member` refusal and the `policy_filtered` marker, which +disclosed hidden members. An inherited placement cannot be dropped or added +per member. This resolves the question of interface-named policy rules: access +is defined by facts on securables, never by rules naming types. From d9caeb69822cef4e0f0fdac240d1e585535244dd Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Mon, 12 Oct 2026 18:35:32 -0700 Subject: [PATCH 4/4] rfc: record Lance 11 behavior and admit cross-member bm25 via a shared scorer Rechecked against the pinned Lance 11.0.0 source and a probe on the prototype branch: - Packed UInt64 endpoint filters (type range, =, IN, values above 2^63) are served by scalar indexes. Endpoint columns use BTREE, since a bitmap loads every distinct key, and packed filters are built from u64 literals. - Row addresses and stable row ids are per dataset, so neither keys rows across member tables. - A handle reopen costs a HEAD because the manifest size is not passed. - bm25 across members becomes a phase 3 feature: one corpus scorer from all members' inverted indexes (build_global_bm25_scorer) injected into each arm (with_base_scorer), replacing the derived projection for cross-type full-text ranking. Verified in source, not yet run. --- docs/rfcs/2026-10-07-polymorphic-types.md | 57 ++++++++++++++++++++--- 1 file changed, 50 insertions(+), 7 deletions(-) diff --git a/docs/rfcs/2026-10-07-polymorphic-types.md b/docs/rfcs/2026-10-07-polymorphic-types.md index 0f1231cc0..c97919f3e 100644 --- a/docs/rfcs/2026-10-07-polymorphic-types.md +++ b/docs/rfcs/2026-10-07-polymorphic-types.md @@ -242,9 +242,11 @@ recurses across Person and Organization with one shortest-distance search. Refusals: -- `bm25` on a binding whose set has more than one member is refused. Lance - computes BM25 statistics per index, so scores are not comparable across - tables. +- `bm25` on a binding whose set has more than one member is refused in + phases 1 and 2. By default Lance builds BM25 statistics from one dataset's + inverted index, so scores from two member tables are not comparable. Phase 3 + admits it with a shared corpus scorer (see Search under Planning and + execution). - `search`, `fuzzy` and `match_text` produce no score; they are membership filters. They run per arm when every member has an inverted index on the property, and are refused, naming the member, otherwise. @@ -314,7 +316,11 @@ query touch($slug: String) { resource bounds. The table-handle cache holds at least every table in the accepted catalog. Today it holds 64, so a 75-member interface evicts its own handles: in the prototype a warm repeat of a 75-member keyed lookup reopened - 55 to 61 tables, against none at 60 members. + 55 to 61 tables, against none at 60 members. Each reopen costs at least one + object-store HEAD on S3: OmniGraph opens a pinned version without passing + the manifest size, so Lance resolves it before consulting its manifest + cache (`lance` `dataset/builder.rs` `resolve_version_location`). Passing + the size recorded in `__manifest` would make a warm reopen free as well. - Every schema migration in this RFC is a schema apply, and schema apply refuses while any branch other than `main` exists. A graph that keeps long-lived branches lands them before widening an endpoint. @@ -373,7 +379,12 @@ identity-derived path. Edge tables already store each endpoint as one packed polymorphic edge table has the same columns, the same indexes and the same row shape as a concrete one. The scalar index on each endpoint column serves `=`, `IN` and the range predicate that selects one endpoint type, -`packed BETWEEN type << 48 AND ((type << 48) | (2^48 - 1))`. +`packed BETWEEN type << 48 AND ((type << 48) | (2^48 - 1))`. That index is a +BTREE. A bitmap index also answers these predicates, but it loads every +distinct key when opened (`lance-index` `scalar/bitmap.rs`), and packed +endpoints are nearly unique. Packed filters are built from `u64` literals: a +value that passes through `i64` loses the index silently, because Lance then +compares through a cast instead of probing. The identity upgrade publishes new table versions for the current snapshot. Retained pre-upgrade versions still carry string endpoints; their decoding, @@ -507,6 +518,17 @@ upgraded current format, and historical abstract bindings remain refused. - **Hydration.** `id_lookup` groups each slice's references by type and probes each member table. `hash_join` builds per member type, or on the packed key. +- **Search across members (phase 3).** `bm25` over an abstract binding builds + one corpus scorer from every member's inverted index with Lance's + `build_global_bm25_scorer`, which sums document counts, token totals and + per-term document frequencies over any list of index handles. Each arm's + full-text plan takes that scorer through `with_base_scorer`, so every arm + scores against the same statistics and scores merge like distances. The + full-text index is declared on the interface property, so every member + shares its analyzer. The scorer must cover every query token, including + fuzzy expansions. Lance's scanner does not expose the hook, so the engine + builds those plans itself. This replaces a derived projection for + cross-type full-text ranking; the API is verified in source, not yet run. - **Search.** `nearest` runs per arm with `k` and merges by `_distance`. The merge preserves each arm's accuracy, because the global top k is a subset of the union of per-arm top-k results. The overfetch ladder runs per arm. @@ -755,7 +777,8 @@ sub-pipelines), no cost-blind plan choice, and no maintained parallel truth. | Join a union's property types over its concrete closure | Matches the alternation rule literally, but adding an implementor to an interface inside a union could change a stored query's result schema. Joining declared member types keeps result schemas independent of membership. | | Refuse a type set that includes a member hidden from the actor (`policy_denied_member`) | Keeps results whole, but discloses that the hidden member exists, and the only escape names the hidden type. Filtering over the actor's schema view with `view.restricted` gives the same completeness signal without disclosure. | | Expand an interface feed filter once, when the cursor is created | Needs no new refusal, but freezes membership: a live view misses every member added later. | -| Normalize BM25 scores across arms, or fuse arms with `rrf` | Normalization is unstable across `k`, filters and pages. Rank fusion over N arms keyed by packed reference is plausible later; today's `rrf` takes exactly two arms and fuses by bare id, which would merge colliding keys. | +| Normalize BM25 scores across arms, or fuse arms with `rrf` | Normalization is unstable across `k`, filters and pages. Rank fusion ignores absolute relevance, and today's `rrf` takes exactly two arms and fuses by bare id, which would merge colliding keys. A shared corpus scorer gives comparable scores directly. | +| A derived interface projection for cross-type full-text search | True global BM25, but a maintained copy of every member's text. Lance 11 already accepts injected corpus statistics, so the same scores come from the members' own indexes. | Prior art. Gel (EdgeDB) keeps a table per concrete type and compiles abstract reads to a per-query `UNION ALL`, with `__type__` as a per-table constant. @@ -793,6 +816,26 @@ column, and of an Overwrite changing row ids and dropping indexes, supported a storage design this revision no longer carries and are not evidence for the integer-identity upgrade. +Lance 11.0.0 behavior this revision relies on was rechecked on 2026-10-12, +by a probe on the prototype branch (`tests/lance_identity_probes.rs`) and in +the pinned source: + +- On a packed `UInt64` column, BTREE and bitmap indexes both serve a type + range (`>=`/`<=` and `BETWEEN`), `=` and `IN`, with correct rows, including + packed values above 2^63 (type id 40000). The parser turns a bounded pair + into `SargableQuery::Range` (`lance-index` `scalar/expression.rs`, + `maybe_range`). +- Row addresses are `(fragment << 32) | offset` per dataset, and stable row + ids also start at 0 in every dataset; two tables produced the same first + addresses. Neither is a key across member tables. +- Each dataset plans index use from its own index metadata, and a table + without the index falls back to a filtered scan. +- Metadata-only column addition: a staged `Operation::Merge` must keep every + original fragment and row count, existing field ids keep their paths, new + ids exceed `max_field_id`, and fragments without the field read null. +- `merge_insert` probes a key-column index with `IsIn` when one exists, which + matches the one data read measured for a keyed insert today. + ### Prototype An end-to-end prototype on branch @@ -999,11 +1042,11 @@ graph. - Interface-scope `@unique` behind `interface-unique`, including full prospective-universe validation on schema changes. Earlier phases refuse its declarations and feature rather than accept unenforced constraints. + - `bm25` across members with a shared corpus scorer. 5. **Phase 4, extensions.** - Covariant interface properties (enum refining String, non-null refining nullable; physically identical). - A top type (`$x: *`, `-> *`) that requires an anchor. - - Derived interface projections for cross-type full-text search. - Downcast access `$x[Person].email`. `implementation` moves to `in-progress` with phase 0 and to `partial` when