feat: Merge v4 into master#70
Open
peterkelly wants to merge 171 commits into
Open
Conversation
Inject Rust-backed libraries through structured virtual-module data instead of generating Rex source and reparsing it. That keeps embedder-provided library interfaces in Rust, which is the right abstraction boundary for ADTs and native exports that already originate as typed host data. Make library-local embedder-defined types visible through imports without requiring a separate global type injection step. The structured path now computes exports and interfaces directly, stores a first-class virtual module record, and injects qualified declarations so imported names resolve the same way they do for source-backed modules. Keep raw Rex declarations as an explicit escape hatch with add_raw_declaration instead of treating them as the default representation. Remove the add_declaration alias, stop caching export declaration strings in the normal structured path, and render virtual-library source only when it is actually needed for debugging or compatibility. Update the embedding documentation and regression tests to cover the intended behavior, including the overlapping-constructor import cases and the library-local embedder-type flow.
Stop treating Rust-side library registration as synthesized Rex source. Injected ADTs and other structured declarations now stay in AST/program form through virtual library resolution, while real Rex source remains reserved for embedder- or user-supplied modules. This keeps the embedding boundary honest and removes source synthesis from the semantics of host type registration.
Make RexAdt registration family-based so embedders can inject a root Rust ADT without manually registering every transitive dependency first. The previous API only injected one declaration at a time, which made acyclic Rust ADT graphs unnecessarily order-sensitive and pushed graph management onto embedders. This change moves that responsibility into the registration layer and keeps the low-level single-ADT primitive for data-driven cases. RexAdt now exposes rex_adt_decl() plus rex_adt_family(), and inject_rex routes through engine support that deduplicates declarations, calculates an acyclic dependency order from variant payload types, and rejects cycles explicitly. Library staging now follows the same family semantics so direct engine injection and library injection behave the same way. The derive macro now emits family discovery automatically for derived Rust types by walking supported field types and collecting reachable derived ADTs. That makes top-level injection sufficient for acyclic families while preserving the existing Rust value conversion code. The tests cover derived dependency closure registration, library-side auto-registration, and explicit cycle rejection. The embedding and crate docs were updated to explain the new family-based behavior and to clarify that cyclic ADT families are still out of scope for this path.
Allow derived ADTs to contain Rust leaf types that participate in Rex value conversion without also forcing those leaf types to be registered as ADT dependencies. The family auto-registration work made dependency discovery treat every non-builtin field type as a RexAdt dependency. That was too strict for leaf types such as AtomRef or Xyzf32 that already implement RexType, IntoPointer, and FromPointer but are not ADTs. In those cases the type mapping is valid, yet the derive macro still tried to recurse into ADT registration and rejected the field. This change adds #[rex(type_only)] as an explicit escape hatch on fields. The attribute preserves the field's RexType mapping and Rust pointer conversions while telling family discovery not to recurse into RexAdt registration for that field. That keeps auto-registration useful for real ADT graphs without conflating every Rex-visible Rust type with an ADT dependency. The derive tests now cover both tuple-style and record-style leaf types, including a BoundingBox example built from Xyzf32 endpoints. The embedding and proc-macro docs were updated so embedders know when to use the new attribute.
The previous design solved auto-registration for acyclic ADT families, but it put the leaf-vs-ADT distinction in the wrong place. Derived ADTs had to mark non-ADT fields with #[rex(type_only)] so the macro would not try to register every user-defined field type as a RexAdt dependency. That was the wrong abstraction boundary. Whether a Rust type contributes ADT declarations is a property of the type, not of each field where it appears. Repeating that information at use sites was easy to forget, easy to misapply, and a poor fit for the goal of making top-level ADT registration ergonomic. This change makes RexType the universal registration surface by adding collect_rex_family with a default no-op implementation. Leaf Rex-visible types now participate in type mapping and pointer conversion without pretending to be ADTs, while true ADT types contribute declarations through this hook. Engine and library registration now operate on T: RexType and collect the family from RexType::collect_rex_family before reusing the existing ordering and cycle checks. RexAdt remains for ADT-specific declaration logic, but family collection now flows through the type-level trait instead of a separate dependency-discovery path. The derive macro now generates RexType::collect_rex_family for derived ADTs and recursively calls collect_rex_family on nested field types. That lets manual leaf types such as AtomRef and Xyzf32 work without any field annotation. The old #[rex(type_only)] support was removed rather than carried forward as a compatibility crutch, because keeping both models would preserve the same conceptual confusion this refactor is meant to eliminate. Manual ADT test fixtures were updated to contribute their declarations from RexType, and the derive regression tests now verify that leaf RexType fields and record fields work without annotations. The docs were updated to match the new model. EMBEDDING.md and the engine and proc-macro READMEs now explain that family registration is driven by RexType::collect_rex_family, that ordinary leaf types inherit the default no-op behavior, and that manual ADTs should override collect_rex_family to add their declarations. This keeps the public guidance aligned with the code and makes the new mental model explicit in the places embedders are most likely to read first.
Move the embedder-facing registration path behind Engine and stop advertising a parallel set of similarly named TypeSystem entry points. Before this change, Engine and TypeSystem both exposed closely related methods for prelude setup and declaration registration, which made the public surface larger than it needed to be and made it easier to pick the wrong abstraction level. Rename the remaining low-level TypeSystem entry points to register_* and new_with_prelude so they read as typing-only operations, while keeping the higher-level inject_* and with_prelude APIs on Engine. This preserves the intended split: TypeSystem owns typing metadata, while Engine owns typing plus runtime installation. The change also removes a few low-value wrapper methods whose behavior was redundant or easy to confuse, with the explicit exception of the raw instance-head registration escape hatch. Update in-tree callers, tests, fuzzers, wasm, LSP code, and embedding documentation to use the renamed TypeSystem APIs where direct type-system mutation is still required. This keeps the workspace consistent with the new naming and makes the distinction between low-level typing setup and embedder-facing runtime setup much harder to miss.
Move more host registration onto Library so embedders have one staged path for anything that can be expressed as a library. Engine now stays focused on materializing staged libraries and on the few genuinely engine-specific escape hatches that are not yet library-shaped. The largest part of this change is teaching global libraries to cover the old direct engine injection cases. Library gained root-scope staging via Library::global(), structured declaration staging, staged ADT families, typed value exports, and the lower-level native export variants with explicit gas cost and cancellation support. Engine::inject_library was updated so a global library can materialize those staged declarations and exports directly into root scope without going through a second public registration API. This also removes the remaining public duplication where it was unnecessary. Direct engine registration helpers that overlapped with library behavior were demoted to internal helpers or removed, and in-tree callers were moved over to staged libraries instead. That includes tests, the LSP, fuzzing entry points, proc-macro generated code, and the CLI embedding surface. Tracing-backed std.io logging now follows the same rule. The public Engine::inject_tracing_log_function APIs are gone; Library now stages those exports directly, and the CLI prelude injects them as part of the std.io library. This keeps logging exports consistent with every other importable host surface instead of sneaking them in through a separate engine-only path. A few engine internals remain on purpose. Raw instance insertion and default-instance synthesis still need engine-local plumbing because they install runtime dictionaries or hidden natives that are not represented cleanly by Library yet. RexAdt::inject_rex also still relies on engine internals for the root-scope derived ADT path, which turned out not to be perfectly equivalent to pure library staging yet. Those are now the exception rather than the model. The docs and tests were updated to match the new shape: stage host declarations and exports on Library, then materialize them with Engine::inject_library. Verification included cargo check, targeted test builds, and a full ./build.sh pass.
Move inference entry points, inference-only helpers, and related tests into rexlang-typesystem/src/inference.rs. Update downstream call sites and docs to use the new free-function inference API while keeping shared type-system registration logic in typesystem.rs.
Add Promise as a reserved unary built-in type constructor without introducing any prelude or runtime functions. This keeps the type available for embedders while preserving the existing value-level surface. Also add regression coverage for reserved-name handling and type annotations, and update the docs that enumerate built-in type constructors.
Add overloaded prelude support for unwrap on Option and Result. The new builtins return the inner value for Some and Ok, and raise an evaluation error for None and Err. Cover the new behavior with engine eval tests and refresh the generated prelude documentation so the documented surface matches the implementation.
Teach the runtime JSON bridge how to encode and decode built-in Promise values using the Promise(uuid) representation. Add regression coverage for both directions of the round-trip with a fixed UUID so promise conversion stays deterministic and verified.
Present library imports and exports as a single name table instead of three user-visible namespaces. A library export can now carry value, type, and class facets under the same spelling, which matches the language model we want users to have and fixes the surprising behavior where item imports only considered values. Restructure LibraryExports around ExportEntry and thread the new shape through import collection, REPL state, rewrite logic, and prelude export construction. Selected and wildcard imports now bind every available facet for a name, while resolution remains context-sensitive so expression, type, and class positions still validate against the correct internal category. Expand the library tests to cover the new semantics as examples, including selected imports, wildcard imports, aliasing, same-name ADT type and constructor imports, class imports, and missing-facet negative cases. Update the language and embedding docs to explain that imports operate on exported names, that a single name may carry multiple facets, and that importing a name does not invent a missing value, type, or class facet.
Keep library qualification in compile-time symbol resolution, but stop carrying those qualified names into runtime ADT values. This makes runtime values match the public ADT surface instead of engine-internal naming. Derived Rust types now round-trip through library injection and low-level JSON conversion without leaking module prefixes into constructor tags or serialized output. Update the engine to allocate bare constructor tags for injected ADTs and to match named patterns by local constructor name. Update the core JSON helpers and std.json primops to encode and decode ADTs using public constructor names rather than qualified internal symbols. Adjust the affected tests to assert the simplified runtime behavior and add an engine library test that exercises derived Rex types in a named library, including serde rename handling for the BAR variant. Confirm the change with ./build.sh.
Add more derives to commonly-used data structures so that they can be ordered and stored in a BTreeMap/BTreeSet. Divide type system code into more files to enhance human readability.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: Currently version no. is at 3.9.1; will bump to 4.0.0 once the API has been fully stabilsed (module/library rename done).
This merge will make it easier for others to start using the new version in the meantime, by just referencing the 3.9.x versions.