Skip to content

Add frozen app schemas, JSON Schema interchange and typed collaboration APIs - #2045

Closed
joepio wants to merge 9 commits into
developfrom
codex/atomic-schema-sdk
Closed

Add frozen app schemas, JSON Schema interchange and typed collaboration APIs#2045
joepio wants to merge 9 commits into
developfrom
codex/atomic-schema-sdk

Conversation

@joepio

@joepio joepio commented Oct 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

App schemas that travel with the data

Apps can define immutable Classes and Properties without hosting an ontology,
edit nested data through Rust/TypeScript APIs, and sync to a cold replica without
pre-registering its schema. Definitions use atomic:frozen:<BLAKE3 of JCS JSON-AD>.
They travel in the resource's atomic:schema-definitions Loro root over existing
COMMIT/UPDATE/SYNC_PUSH paths, including Iroh.

Implementation

  • Verify immutable bodies, bounded dependency closures and instance data before
    persistence. Install reachable definitions with admitted data; missing
    dependencies fail explicitly. Verify unused attachments too. No schema network
    fetch or peer-supplied code execution is introduced.
  • Limit the visible map to 512 definitions / 1 MiB, 256 KiB per body, depth 16.
    Exact-body verification caches are bounded and cannot supply missing definitions
    across stores. These are post-import limits, not a CRDT decompression DoS audit.
  • Provide shared Property bindings, identity-preserving aliases, nested/list
    edits, enum/nullable/union shapes, generated Rust/Dart models, and bounded JSON
    Schema 2020-12 import/export with verified identity sidecars.
  • Provide trusted native copy migrations with preview, source/version pins,
    resumable progress and retained originals. Receiving schemas never runs a
    migration; concurrent old-version edits require explicit reconciliation.
  • Withhold browser sync completion after invalid data, worker-write or flush
    failures. Errors identify the rejected resource.

Review fixes

  • Attach schemas directly to pending native documents before commit messages are
    assigned. Genesis creator attribution and distinct signed edit tokens are
    covered by a regression that failed on the reviewed draft.
  • Remove native snapshot round trips and duplicate dependency resolution.
  • Browser admission reads only schema-relevant values and reuses one isolated
    validation replica, freed after five idle seconds. Local changes refresh it;
    rejected or abandoned candidates cannot affect later checks. A previously
    ordinary resource is still checked when an incoming delta introduces a schema.
  • Synthetic large-canvas repeated-import median: reviewed draft 197 ms,
    corrected path 14 ms, pre-PR path 14.5 ms. First imports, rendered UI
    and Android are not covered by this benchmark. Details and reproduction:
    review follow-up.

Compatibility and scope

This remains a draft. String tagging, preserved nested JSON nulls and stricter
Rust setters also affect resources without app schemas. Older readers may still
interpret JSON-like strings differently. Upgrade readers and writers together;
there is no negotiated schema-enforcement capability. Core HTTP ontology IDs stay
unchanged. Existing frozen definitions cannot be edited; semantic changes need a
new definition and migration.

The mutable plugin-schema API remains separate, and typed nested/list edits are
not exposed through plugin RPC. Plugin standardization requires coordination.
A standalone Dart transport/editing client, browser migration orchestration and
unrestricted JSON Schema support remain follow-up work. The draft retains its
review history; representation, protocol/admission and SDK layers still need
explicit review before merge.

Validation

  • Native: 704 library tests passed, 8 ignored, plus 7 generated-model/protocol
    integration tests. The 25 schema tests also passed after adding resource-subject
    diagnostics. An initial run hit the existing macOS compaction test failure;
    both its isolated rerun and the full suite rerun passed.
  • Browser unit suite: 1,150 passed, 2 skipped (existing live test and opt-in
    performance probe). Live-server integration tests excluded.
  • TypeScript typecheck, full recursive browser lint, rustfmt, workspace Clippy
    with warnings denied, and four Dagger/fixture setup checks pass. The commit
    hook also checks the exact staged snapshot.
  • Coverage includes tampered/oversized/missing definitions, cold retrieval,
    authorization, migration races, replica expiry/local edits/failed candidates,
    signed attribution, and valid browser entries following rejected ones without
    false sync completion.

Atomic Audio adoption (separate repository)

Audio backport: d13c280632c6842c792bbd328223b8936e4f5f00.
Validation and boundaries,
vendored signing regression,
Mac audio counters.

Audio passes 29 native tests, 43 Flutter tests, analysis, all-target Clippy and
formatting. Full Mac integration passes in 74 seconds with zero underruns,
deadline misses or device errors across 54,004 callbacks. The normal Mac app was
rebuilt afterwards. Android hardware and a deployed phone/browser round trip
were not retested. These results belong to the linked Audio commit, outside this diff.

CI status

Exact-head run for efca818a318147d37ec5325c25354fa91e151854
was dispatched on 2026-10-05. Main (Mancave) and downstream atomic-saas are queued
as of this update. This draft is not merge-ready without a concluded green run;
local checks do not replace CI or establish SaaS compatibility.

@joepio joepio changed the title Add frozen app schemas and nested field editing Add frozen app schemas with verified sync and nested field editing Oct 4, 2026
@joepio joepio changed the title Add frozen app schemas with verified sync and nested field editing Add frozen app schemas, typed models and explicit collaboration APIs Oct 4, 2026
@joepio joepio changed the title Add frozen app schemas, typed models and explicit collaboration APIs Add frozen app schemas, JSON Schema interchange and typed collaboration APIs Oct 4, 2026
@joepio
joepio force-pushed the codex/atomic-schema-sdk branch from efca818 to daef1c5 Compare October 5, 2026 09:34
@joepio Claude

joepio commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

Review after discussing the direction with Joep: please don't merge this as one PR. The parts are useful, but the front door duplicates the existing ontology model.

Direction

There is one schema model: Ontology, Class and Property, edited in the existing ontology editor. Every API (code, JSON Schema, LLM tools) writes ordinary ontologies. The building blocks already exist: ensureSchema in browser/lib/src/plugin-schema.ts (code-first and idempotent) and @tomic/cli (typed code from ontologies).

Identity

Every Ontology, Class and Property keeps its live ID (HTTP or atomic:), and data keys always use that live ID. The server derives a frozen ID from the content, like lastCommit:

  • It is BLAKE3 over canonical JSON, as in this PR.
  • It hashes only the fields that carry meaning (datatype, classtype, allowsOnly, constraints, shortname and scope), not names or descriptions.
  • It is hashed bottom-up: a Class covers its Properties' frozen IDs, and an Ontology covers its Classes'.

Old versions live in the Loro history, indexed by frozen ID. A frozen ID is shared together with its live ID, so it can be resolved.

What to keep from this PR, each as its own PR

  1. The string tag and JSON null storage changes (these affect all data).
  2. JSON Schema import and export, writing an ordinary ontology.
  3. Shape constraints as optional properties on ordinary Properties.
  4. The nested and list edit APIs.
  5. The hashing code, reused for the derived frozen ID.
  6. The migration runner, for explicit bulk migrations.

Drop: defineAppSchema, registerAppSchema, setAppField, bindings and compose, plus the separate TypeScript, Rust and Dart generators. Generate code from ontologies in @tomic/cli instead.

Later, and only if Audio or another app needs strict offline checks: carrying definitions with data and checking them on arrival. Validation should apply to writes, never block a drive's sync over one resource.

Apps ship a bundle of their ontology with fixed live IDs, made once by the developer, and import it on startup. Each install making its own ontology wouldn't work, because device IDs would diverge.

Generated by Claude Code

@joepio joepio mentioned this pull request Oct 5, 2026
Closed
@michielbdejong michielbdejong mentioned this pull request Oct 8, 2026
Merged
@joepio joepio mentioned this pull request Oct 8, 2026
Merged
3 tasks done
@michielbdejong michielbdejong mentioned this pull request Oct 8, 2026
Open
joepio added a commit that referenced this pull request Oct 9, 2026
@joepio
Schemas: content-addressed properties, class constraints, JSON Schema…
00031b4 
… and lenses (#2151)

* Plan class constraints and their effect on forms

Allowed values, limits and linked classes move from Properties to a
JSON Schema-style constraints map on the class, so properties can become
immutable. Covers what changes for forms, tables and kanban, and why this
lands separately from #2045.

* Specify content-addressed property identity

A Property's ID becomes atomic:prop:{blake3-hex}, the BLAKE3 derive-key
hash (context "atomic property identity v1") of the JCS object
{datatype, ontology, shortname}. Adds the spec page with test vectors,
property_id / verify_property_id in atomic_lib, and the atomic:prop:
identifier kind. Nothing creates or enforces these IDs yet.

* Mint content-addressed property IDs in @tomic/lib

Adds propertyId / verifyPropertyId (same vectors as atomic_lib), the
property identifier kind, and a contentAddressedProperty option on
Store.newResource that derives the subject from parent, shortname and
datatype instead of minting a genesis cert. Property keys written as
did:ad:prop: are stored under their atomic:prop: form.

* Create properties under content IDs and keep their identity fixed

Table columns, select columns, form fields, the ontology editor and
ensureSchema now create Properties with contentAddressedProperty, only
once parent, shortname and datatype are final. Renaming a column or form
field changes its label, not its shortname. Changing the datatype of a
content-addressed property creates a new property and swaps it into its
classes (existing values move once lenses land). Re-parenting such a
property is refused. Legacy properties keep editing in place.

* Accept and protect content-addressed properties in commits

A genesis commit for an atomic:prop: subject is valid when the ID is the
hash of its parent, shortname and datatype; it needs no genesis cert and
append rights on the parent. A repeat genesis of the same ID from
another device merges. Later commits cannot change a property's parent,
shortname or datatype.

* Add class constraints: JSON Schema keywords on a Class, enforced on commit and save

A Class gets a `constraints` map (property subject to enum, minimum,
maximum, exclusive bounds, min/maxLength, min/maxItems, pattern, class).
Rust and TS share one fixture. Enforced at commit under validate_schema and
in Resource.save for loaded classes; peer commits and reconcile skip it.

* Add ensureOntology and JSON Schema import/export

ensureOntology finds or creates an Ontology with content-addressed
properties, classes, requires/recommends and class constraints; running it
twice writes nothing. ensureSchema becomes a deprecated alias.
ontologyFromJsonSchema / ontologyToJsonSchema map draft 2020-12 onto that
input and back; unsupported constructs fail with a JSON pointer. Rust twin
in lib/src/schema/json_schema.rs (import + ensure_ontology), sharing one
fixture with the TS tests.

* Add schema push, lock and check to atomic-cli and @tomic/cli

Both tools turn a JSON Schema or ontology file into a real ontology and
write <file>.lock.json pinning the property IDs. lock works offline,
check fails when a pinned property ID changed (a new shortname or datatype
makes a new property), push refuses a stale lockfile unless
--accept-changes. Rust and TS write byte-identical lockfiles, proven by a
shared fixture.

* Move tables, kanban and the ontology editor to class constraints

Select options, single pick, number/text limits and linked classes are
written to the table class's constraints map instead of the Property.
Readers use getEffectiveConstraint: the class map wins per keyword, legacy
Property allowsOnly/max/classtype is the fallback, so existing tables keep
working. The ontology editor edits constraints per property in each class
card; content-addressed Property cards no longer edit those fields. The
form builder keeps its property-level writes until its own step.

* Let an empty allowsOnly restrict nothing

A SelectProperty must carry allowsOnly, but its options now live in the
class constraints map, so createSelectProperty stores []. Resource::set
read that as 'allows nothing' and every choice submission failed.

* Move forms to class constraints

The form builder reads and writes enum, maxItems and class on the form's
data class. Form options use JSON Schema keywords (minimum/maximum,
minItems/maxItems; old names still read) and may only tighten the class
constraint. forms.rs and form-renderer validate submissions against the
narrowed constraints, and describe_form returns the narrowed JSON Schema.

* Save a recreated property even after its genesis is signed

Signing the genesis clears `new` while the commit waits for save(), so
a renamed form field's sibling property was never committed and the
table could not load the column after a reload. Choice options in the
class constraints now follow the field to its new property.

* Give the assistant find_schema and ensure_ontology

find_schema searches the drive's classes by words and returns each match's
JSON Schema, replacing get_user_classes. ensure_ontology takes a JSON
Schema, imports it with ontologyFromJsonSchema and runs ensureOntology;
import errors come back verbatim with their JSON pointer, and a second call
changes nothing. Skills tell the model to search first and reuse classes.

* Add lenses: content-addressed mappings that keep old data readable

A Lens (atomic:lens:{hash} of from, to, transform) maps one property onto
another with rename, wrap, head, map or convert. atomic_lib and @tomic/lib
derive the missing side when a resource materializes; the signed Loro doc
never changes and real values always win. Lenses are owned by the target
property's ontology, are backfilled into stored resources when they land,
and never reject a commit. Changing a property's datatype or a form field's
shortname now creates the matching lens.

* Format lens.test.ts

* Mount lib/tests/fixtures in the CI JS test container

The class constraint, lens, JSON Schema and lockfile tests read fixtures
shared with their Rust twins by repository-root path.

* Fix select columns, linked classes and template tags on the new schema model

- Tags for a hosted (atomicdata.dev) property get a DID subject under the
  table's row class; the server rightly refused path subjects there, which
  broke template drives.
- A draft select column no longer saves its Property early; tags created
  in the draft reach the class enum.
- Select cells wait for loading tag options before picking; first-row
  constraints include the table's row class.
- The property dialog offers a Linked class field for link datatypes,
  writing the class's class constraint.
- Lenses and recreated properties are always saved (signing the genesis
  already clears new); views swap old columns for the new property.
- The generic edit page keeps a content-addressed property's identity
  read-only; the form builder says why a looser limit is refused.

* Changelog entries for the schema change

* Set validate_constraints in the sync ownership test from develop

* Return the effective constraint from the store snapshot

The React Compiler (data-browser compiles @tomic/react) cached the
constraint computed from the hook's unchanging arguments, so the ontology
constraint editor kept showing the value it first read after every save.
The constraint is now the useSyncExternalStore snapshot, rebuilt after each
change. Vitest compiles browser/react/src with the React Compiler so the
regression test sees the same code the app runs.

* Give the hosted MCP find_schema and ensure_ontology

find_schema searches the classes on the drives the agent can read and
returns each as JSON Schema; get_user_classes is now an alias for it.
ensure_ontology imports a JSON Schema and creates the ontology with commits
signed as the issued agent through the normal pipeline, offered only when
editing was allowed. atomic_lib gains ontology_to_json_schema, checked
against the same fixture as the TypeScript export. MCP class and property
arguments accept atomic: subjects, and a class's own name property no
longer clashes with the standard one.

* Tidy the class constraints section and form bound fields

The Constraints toggle is now a small text button with a summary when
collapsed, and its inputs sit on a grid that shares the right edge of the
add-property row (full width when the row wraps). Min/max pairs in the form
builder align to the top, and a bound's error shows in flow, fully visible.
@joepio Claude

joepio commented Oct 10, 2026

Copy link
Copy Markdown
Member Author

Complexity review (about 9.7k changed lines; I read dependencies.rs and schema-dependencies.ts side by side and only skimmed the other schema files; tests, fixtures and docs not read).

  1. The schema system is implemented twice, in Rust and in TS (plus a Dart copy in model_runtime.dart). Pairs by size: lib/src/schema/json_schema.rs (577) vs browser/lib/src/schema-json-schema.ts (541), dependencies.rs (360) vs schema-dependencies.ts (283), shape.rs (217) vs schema-shape.ts (263), frozen.rs (75) vs schema-frozen.ts (95), app.rs (255) vs app-schema.ts (260). Both sides hand-code the same limits (512 definitions, 1 MiB, depth 16, 128 fields, 256 cache entries). Every limit or rule change is two edits that can drift. Prefer having the TS side call the WASM build for validation and canonicalization over keeping a parallel checker; the interop fixture then becomes a smoke test instead of the only drift guard.
  2. The verified-definition memo cache (dependencies.rs ~85-141) scans a VecDeque of up to 256 bodies and compares full bodies on every lookup, behind a global OnceLock<Mutex>, to save parsing a body of at most 1 MiB. Delete it, or key it on a content hash. Verification still runs on a miss, so behaviour stays the same.
  3. Link discovery differs between the sides: TS (schema-dependencies.ts links()/classIds() ~28-79) handles an isA that arrives as a JSON-array string; Rust (references() ~22-56) does not. That is a latent behaviour mismatch, and merging the two implementations would settle which one is right.
  4. Bindings: defineAppProperty and bindAppProperty look close to one function; schema-bindings.ts (177) plus bindings.rs (201) plus the Dart runtime is four near-identical typed wrappers. I did not read these bodies, so check before merging them.

Separately, this adds a second schema API next to ensureSchema and the ontology editor; the standing call from 5 Oct is that any schema API writes ordinary ontologies the editor shows. That is the bigger complexity question for this PR.

Generated by Claude Code

@joepio joepio mentioned this pull request Oct 10, 2026
Merged
2 of 3 tasks
@joepio Claude

joepio commented Oct 10, 2026

Copy link
Copy Markdown
Member Author

Closing: #2151 (content-addressed properties, class constraints, JSON Schema and lenses, merged 9 Oct) replaced this PR and took over its JSON Schema conversion code. The string tag and JSON null fixes, which affect all data and which #2151 did not carry over, are now in #2206.

Complexity Critic's review: the bindings, the dependency memo cache and the Rust/TS link-discovery mismatch (to_subjects errors on an isA that arrives as a JSON-array string, which TS accepts) all live in code that is no longer going in, so they are settled by not merging it. The nested/list edit APIs and the migration runner stay on this branch in case Atomic Audio needs them; the branch is not deleted, because Audio vendors it.

Generated by Claude Code

@joepio joepio closed this Oct 10, 2026
@joepio joepio mentioned this pull request Oct 11, 2026
Merged
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@joepio