diff --git a/CHANGELOG.md b/CHANGELOG.md index 19d96cea17..44ee1f2a8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,9 @@ See [STATUS.md](server/STATUS.md) to learn more about which features will remain ## UNRELEASED +- Schema is recommended, not required: `Resource::set` no longer fails when + the Property resource is missing. Datatype and `allowsOnly` still apply when + the Property exists. See [`planning/optional-schema.md`](./planning/optional-schema.md). - The outbox drains over a live Iroh link too (`sync::peer::LivePeerCommitTransport`): a device with no hub in reach delivers its queued writes to a paired peer as signed `COMMIT` frames, which the peer validates and applies like a hub diff --git a/TESTING_COVERAGE.md b/TESTING_COVERAGE.md index 6bee8ca6b0..9e4c4b4ead 100644 --- a/TESTING_COVERAGE.md +++ b/TESTING_COVERAGE.md @@ -766,6 +766,22 @@ Not covered: derived AI tools invoked through a real model; MCP protocol project Not covered: visual morph of a grid card into the resource page in Firefox (needs a headed Firefox run; Playwright's firefox project is locks-only and automation bypasses view transitions unless `forceViewTransitions` is set). +## Schema optionality + +Policy: [`planning/optional-schema.md`](./planning/optional-schema.md). Schema is +recommended, not required on the write path. + +| Flow | Layer | Where | +|---|---|---| +| Classless resource + unknown Property URL saves and reloads | protocol | `lib/src/resources.rs::set_accepts_unknown_property_on_classless_resource` | +| Known Property still rejects a datatype mismatch | protocol | `lib/src/resources.rs::set_still_enforces_datatype_when_property_exists` | +| Unresolvable `isA` does not fail `check_required_props` | protocol | `lib/src/resources.rs` (existing class-skip test) | +| `@tomic/lib` `set` skips validation when `getProperty` fails | glue | `browser/lib/src/resource.ts` (warn + write); no dedicated assertion | + +Not covered: JSON-AD parse of a key with no Property resource (still fails +unless `skip_unknown_props`); a browser integration test that saves a +classless custom-property resource through a real server. + ## Documents | Flow | Layer | Where | diff --git a/docs/src/core/concepts.md b/docs/src/core/concepts.md index faf2519451..23c4a9d01c 100644 --- a/docs/src/core/concepts.md +++ b/docs/src/core/concepts.md @@ -6,6 +6,7 @@ Atomic Data is a modular specification for sharing information on the web. Since Atomic Data is a _modular_ specification, you can mostly take what you want to use, and ignore the rest. The _Core_ part, however, is the _only required_ part of the specification, as all others depend on it. +[Atomic Schema](../schema/intro.md) (Classes, Property resources, required fields) is recommended for apps that want typed forms and shared meaning, but it is not required to store or sync data. Atomic Data Core can be used to express any type of information, including personal data, vocabularies, metadata, documents, files and more. It's designed to be easily serializable to both JSON and linked data formats. diff --git a/docs/src/js-lib/resource.md b/docs/src/js-lib/resource.md index c2bd5842b9..cd9b140e5b 100644 --- a/docs/src/js-lib/resource.md +++ b/docs/src/js-lib/resource.md @@ -108,7 +108,7 @@ By default, `.set` validates the value against the properties datatype. You should await the method when validation is enabled because the property's resource might not be in the store yet and has to be fetched. > [!NOTE] -> Setting validate to false only disables validation on the client. The server will always validate the data and respond with an error if the data is invalid. +> Setting `validate` to false only disables the client-side datatype check. If a Property resource exists (locally or on the server), its datatype and `allowsOnly` are still enforced when that Property can be resolved. Unknown Property URLs are accepted: schema is recommended, not required. A Class's `requires` is checked only when the resource has a resolvable `isA`. **Parameters** diff --git a/docs/src/js-lib/store.md b/docs/src/js-lib/store.md index 50a1840cee..10a13ff118 100644 --- a/docs/src/js-lib/store.md +++ b/docs/src/js-lib/store.md @@ -94,7 +94,7 @@ It takes an options object with the following properties: |----------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------| | subject | string | **(optional)** The subject the new resource should have, by default a random subject is generated | | parent | string | **(optional)** The parent of the new resource, defaults to the store's `serverUrl` | -| isA | string \| string[] | **(optional)** The 'type' of the resource. determines what class it is. Supports multiple classes. | +| isA | string \| string[] | **(optional)** The class of the resource. Recommended — enables required fields, generated types, and generic forms. Not required to save. | | propVals | Record | **(optional)** Any additional properties you want to set on the resource. Should be an object with subjects of properties as keys | ```typescript diff --git a/docs/src/js.md b/docs/src/js.md index ada2c980ab..91f44169ae 100644 --- a/docs/src/js.md +++ b/docs/src/js.md @@ -64,6 +64,19 @@ const newResource = await store.newResource({ await newResource.save(); ``` +A class is recommended, not required. This also works — you lose generated types and generic forms, not persistence: + +```ts +const note = await store.newResource({ + propVals: { + [core.properties.name]: 'Buy milk', + 'https://example.com/done': false, + }, +}); + +await note.save(); +``` + ### Subscribing to changes ```ts diff --git a/docs/src/schema/faq.md b/docs/src/schema/faq.md index 9a226f4476..aedd33a2b5 100644 --- a/docs/src/schema/faq.md +++ b/docs/src/schema/faq.md @@ -1,6 +1,14 @@ {{#title Atomic Schema FAQ}} # Atomic Schema FAQ +## Do I have to define an ontology before I can store data? + +No. Classes and Properties are the recommended way to describe your model, but they are not a write-path requirement. + +A Resource is a subject plus property → value pairs. Property *keys* are URLs. A Property *resource* (shortname, datatype, description) is optional. A resource with no `isA` has no required properties. Commits and sync work either way. + +What you lose without a schema: required-field checks, shortnames, generated TypeScript/Rust types, generic forms and tables, and a shared meaning other apps can reuse. The easy path is still to declare an Atomic Schema — in code, once that API exists, or in the Ontology editor. The store will not reject a write just because you have not done that yet. + ## Do you have an `enum` datatype? There is no dedicated `enum` datatype but you can use the `allows-only` property to achieve the same effect. diff --git a/docs/src/schema/intro.md b/docs/src/schema/intro.md index e8048f95a6..5e94b85d8b 100644 --- a/docs/src/schema/intro.md +++ b/docs/src/schema/intro.md @@ -6,6 +6,8 @@ You can compare it to UML diagrams, or what XSD is for XML. Atomic Schema deals with validating and constraining the shape of data. It is designed for checking if all the required properties are present, and whether the values conform to the datatype requirements (e.g. `datetime`, or `URL`). +Atomic Schema is **recommended, not required**. [Atomic Data Core](../core/concepts.md) is enough to store and sync resources: property keys are URLs, values are typed, commits are signed. You do not need to publish Classes or Property resources before writing data. Schema is what you add when you want required fields, shortnames, generated types, generic forms, and shared meaning. Libraries accept writes without a schema and enforce a schema when one is present. + This section will define various Classes, Properties and Datatypes (discussed in [Atomic Core: Concepts](../core/concepts.md)). ## Design Goals diff --git a/lib/src/resources.rs b/lib/src/resources.rs index f3204caedd..b9deb56910 100644 --- a/lib/src/resources.rs +++ b/lib/src/resources.rs @@ -1393,16 +1393,33 @@ impl Resource { } /// Inserts a Property/Value combination. - /// Checks datatype. /// Overwrites existing. /// Adds the change to the commit builder's `set` map. + /// + /// Schema is recommended, not required. When a Property resource exists, + /// its datatype and `allowsOnly` are enforced. When it does not, the + /// `Value`'s own datatype is trusted — the same write is already possible + /// with `set_unsafe` and on the Loro commit path. See + /// `planning/optional-schema.md`. pub async fn set( &mut self, property: String, value: Value, store: &impl Storelike, ) -> AtomicResult<&mut Self> { - let full_prop = store.get_property(&property).await?; + let full_prop = match store.get_property(&property).await { + Ok(prop) => prop, + Err(e) => { + tracing::debug!( + property = %property, + subject = %self.get_subject(), + error = %e, + "No Property resource; writing value without schema check" + ); + self.set_unsafe(property, value)?; + return Ok(self); + } + }; if let Some(allowed) = full_prop.allows_only { let error = Err(format!( "Property '{}' does not allow value '{}'. Allowed: {:?}", @@ -1989,6 +2006,46 @@ mod test { assert_eq!(found_prop.to_string(), value.to_string()); } + /// An app can persist data without publishing Classes or Properties. + /// Schema is recommended, not required — see `planning/optional-schema.md`. + #[tokio::test] + async fn set_accepts_unknown_property_on_classless_resource() { + let store: crate::Db = init_store().await; + let unknown = "https://example.com/properties/customTitle"; + let mut resource = Resource::new_generate_subject(&store).unwrap(); + resource + .set(unknown.into(), Value::String("Buy milk".into()), &store) + .await + .unwrap(); + resource + .set(urls::NAME.into(), Value::String("Note".into()), &store) + .await + .unwrap(); + let subject = resource.get_subject().clone(); + resource.save_locally(&store).await.unwrap(); + + let loaded = store.get_resource(&subject).await.unwrap(); + assert!( + loaded.get(urls::IS_A).is_err(), + "classless write must not invent an isA" + ); + assert_eq!(loaded.get(unknown).unwrap().to_string(), "Buy milk"); + } + + #[tokio::test] + async fn set_still_enforces_datatype_when_property_exists() { + let store: crate::Db = init_store().await; + let mut resource = Resource::new_generate_subject(&store).unwrap(); + let err = resource + .set(urls::NAME.into(), Value::Integer(1), &store) + .await + .unwrap_err(); + assert!( + err.to_string().contains("did not match"), + "known Property must still reject a datatype mismatch, got: {err}" + ); + } + #[tokio::test] async fn push_propval() { let store: crate::Db = init_store().await; diff --git a/planning/README.md b/planning/README.md index 7233cc573c..4fa8e14f35 100644 --- a/planning/README.md +++ b/planning/README.md @@ -112,7 +112,8 @@ browser flow; standalone recovery remains self-managed. | [`commit-retention-and-state-certificates.md`](./commit-retention-and-state-certificates.md) | **Superseded.** Envelope-on-resource replaced the retention design; Phase 1 and 2.5 shipped. Only the `stateHash` certificate and per-resource `retention` propval remain from this document. | | [`s3-blob-storage.md`](./s3-blob-storage.md) | **Partial.** Server-wide S3, verified migration and SaaS enforcement implemented. The backend trait is `get/put/size` only: streaming, delete/GC, per-tenant configuration and encrypted Vault attachments remain. | | [`plugins.md`](./plugins.md) | **Partial, off `develop`** — one plugin model (`run` end to end, per-app agents, unattended runs). The code lives on `feat/plugin-model` (PR #1307, 532 files, conflicts with `develop`) and #1482. On `develop` the plugin RPC still answers "not implemented". | -| [`json-schema-code-first.md`](./json-schema-code-first.md) | **Proposal**; nothing on `develop`. `defineSchema` + frozen `did:ad:` schemas are in PR #1262, a draft last touched 2026-09-01. | +| [`optional-schema.md`](./optional-schema.md) | **Decision.** Apps can store and sync without Classes/Properties. Nudge toward Atomic Schema; do not force it on the write path. | +| [`json-schema-code-first.md`](./json-schema-code-first.md) | **Proposal**; nothing on `develop`. `defineSchema` + frozen `did:ad:` schemas are in PR #1262, a draft last touched 2026-09-01. The recommended on-ramp, not a requirement — see [`optional-schema.md`](./optional-schema.md). | | [`android-data-reuse.md`](./android-data-reuse.md) | **Draft.** One store/agent/Iroh node per Android device. Nothing built. Supersedes `on-device-atomic-daemon.md`. | | [`SDK-API-design.md`](./SDK-API-design.md) | SDK / agent DX direction. | | [`api-plugins.md`](./api-plugins.md) | **Exploratory, off `develop`** — rebuilding PR #1383 (OpenAPI/OAuth imports) on the plugin model. LocalThought catalog/connect and Syncables typed imports are implemented on `codex/localthought-api-plugins`; live verification awaits proxy #25. | diff --git a/planning/SDK-API-design.md b/planning/SDK-API-design.md index a90969ab3e..77bf039034 100644 --- a/planning/SDK-API-design.md +++ b/planning/SDK-API-design.md @@ -27,7 +27,7 @@ In the "old" HTTP based Atomic(Server) UX, an app developer had to: ## Future situation - Easy to follow end-to-end tutorial -- Schema creation in-code (no need to use the Ontology Editor if you're just writing code). See [`json-schema-code-first.md`](./json-schema-code-first.md). +- Schema creation in-code (no need to use the Ontology Editor if you're just writing code). See [`json-schema-code-first.md`](./json-schema-code-first.md). Schema is the recommended path, not a write-path requirement — see [`optional-schema.md`](./optional-schema.md). - We provide not just the pipework for persistence, sync, authentication, authorization, but also useful front-end components to provide a unified and secure experience ## What needs to happen diff --git a/planning/json-schema-code-first.md b/planning/json-schema-code-first.md index 93141d3672..4356208f7c 100644 --- a/planning/json-schema-code-first.md +++ b/planning/json-schema-code-first.md @@ -15,6 +15,11 @@ schema is the write-path policy; #1251 becomes a frozen ontology Make Atomic usable by app developers who want to define their data model in code, without first publishing Classes and Properties at HTTP URLs. +This is the recommended on-ramp, not a requirement. Persistence and sync +already work with no schema — see [`optional-schema.md`](./optional-schema.md). +`defineSchema` exists so choosing Atomic Schema is easy, not so skipping it +is forbidden. + The desired workflow: 1. An app declares a JSON Schema-like model in TypeScript, Rust, or another SDK. diff --git a/planning/optional-schema.md b/planning/optional-schema.md new file mode 100644 index 0000000000..e1c9aa631c --- /dev/null +++ b/planning/optional-schema.md @@ -0,0 +1,145 @@ +# Optional schema: nudge, don't force + +**Status:** Decision. Rust `Resource::set` no longer requires a Property +resource. Remaining force-points listed below. + +## The question + +Can someone build an app on `atomic_lib` without specifying an ontology +(Classes and Properties)? If not, should we change that? + +## Answer + +**Yes.** Persistence, commits, and sync already work without Classes or +Properties. Atomic Schema is the recommended path, not a write-path +requirement. + +An app can create resources, set URL-keyed properties, sign commits, and +sync them over WebSocket or Iroh with no `isA` and no Property resources +in the store. What they lose is the *product* of schema: required-field +checks, shortnames, generated types, generic forms/tables, query UX, and +cross-app meaning. + +## Policy + +1. **Atomic Core is enough to store and sync.** A Resource is property → + value pairs with a subject. Property *keys* are URLs. A Property + *resource* (shortname, datatype, description) is optional. +2. **When schema is present, enforce it.** A resolvable Class's + `requires` must be present. A resolvable Property's datatype and + `allowsOnly` must match. That is the contract the author published. +3. **When schema is absent, accept the write.** An unknown Property is + not invalid. An unresolvable Class is skipped (already the + `get_classes` rule — a store cannot enforce a contract it does not + have). A resource with no `isA` has no required properties. +4. **Nudge toward Atomic Schema in DX, not in the commit gate.** The + easy, documented path is still to declare a schema. Code-first + [`defineSchema`](./json-schema-code-first.md) is the on-ramp that + removes HTTP-ontology friction. Tutorials, generated types, and the + Data Browser generic UI should make schema feel like the default. +5. **Do not reject commits for unknown properties.** `validate_schema` + means "check this resource against the Classes it claims", not + "every key must resolve to a Property resource". + +This matches Atomic Data Core vs Atomic Schema in the spec: Core is +required; Schema is a module. See +[`docs/src/core/concepts.md`](../docs/src/core/concepts.md) and +[`docs/src/schema/intro.md`](../docs/src/schema/intro.md). + +## What already worked without schema + +- Classless resources. Internal tests already create them on purpose + (`lib/src/sync/tests.rs` "Classless (no `isA`) so there are no + required-property schema constraints"). +- `@tomic/lib` `Resource.set`: if `getProperty` fails, validation is + skipped and the value still lands in Loro. +- Flutter `set_property` already calls `set_unsafe`. +- Loro `datatypes` tags come from the `Value` variant, not from a + Property lookup. Load-bearing types survive a round-trip without a + Property resource. +- JSON-AD serialize (`propvals_to_json_ad_map`) does not fetch + Properties. +- `check_required_props` is a no-op when `isA` is missing or every + class is unresolvable. + +## What still forced schema (and what we changed) + +| Surface | Was | Now | +| --- | --- | --- | +| Rust `Resource::set` | `get_property` hard-failed | If the Property resource is missing, trust the `Value` and write. If it exists, still enforce datatype + `allowsOnly`. | +| Docs / SDK examples | Ontology-first only | FAQ + JS getting-started show a classless write. Schema examples stay the default. | + +## Remaining force-points (nudge surfaces, leave for now) + +These need schema to do their job. That is a nudge, not a write-path +gate. Do not "fix" them by inventing implicit Properties. + +| Surface | Why it needs schema | +| --- | --- | +| `set_string` / `set_shortname` | Must look up datatype / resolve a shortname. | +| JSON-AD *parse* | Chooses the `Value` variant from the Property datatype. Loro commits do not go through this path. | +| JSON-LD / pretty JSON export | Shortnames and `@context` come from Property resources. | +| `resource.props` / `@tomic/cli` types | Generated from Classes. Untyped `.get(url)` works without them. | +| Data Browser forms, tables, query builder | Columns, inputs, and filters are schema-driven. A classless resource still opens; you get a generic property list. | +| HTTP ontology + `@tomic/cli` workflow | The current documented app-builder path. Replaced by [`json-schema-code-first.md`](./json-schema-code-first.md), not by dropping schema. | + +JSON-AD parse is the one import-path sharp edge: posting JSON-AD with a +key that has no Property resource fails unless `skip_unknown_props`. +Commits (`loroUpdate`) do not have this problem. Softening parse to +infer from JSON (string/number/boolean/array/object) is a later change; +do not block app builders on it — they should use `set` + `save`. + +## How to nudge + +Keep these as the happy path. None of them should become required. + +- **Code-first schema.** `defineSchema` / `store.registerSchema` produces + local DID-backed Class and Property resources. See + [`json-schema-code-first.md`](./json-schema-code-first.md). This is + how we want app developers to *start*, once it exists — not HTTP + ontology editor + CLI export. +- **Generated types.** Annotating `getResource()` is the reward + for having a schema. +- **Generic UI.** Forms, tables, and the assistant get better as soon as + Classes exist. A schemaless app can still render its own views. +- **Docs.** Lead examples use `isA` and ontology objects. One + classless example exists so "do I have to?" is answered with "no, + but you will want to." +- **Shortnames.** `description` instead of a URL only works when a + Property resource is in the store. + +## How not to nudge + +- Do not require `isA` on `newResource` / `create_resource`. +- Do not require Property resources to exist before `set`. +- Do not require an HTTP-hosted ontology. +- Do not invent a second schemaless key format (plain `"title"` keys). + Property identity stays a URL. An app that skips schema still uses + URL keys — `https://example.com/title` or a `did:ad:` Property they + create later. That keeps the data Atomic-Core-valid and upgradeable + to a schema without rewriting subjects. + +## Relation to other plans + +- [`json-schema-code-first.md`](./json-schema-code-first.md) is the + on-ramp for people who *want* a schema without publishing HTTP + Classes. This doc is the rule for people who do not want one yet. +- [`SDK-API-design.md`](./SDK-API-design.md) should keep "schema in + code" as the recommended tutorial path, and mention that schema is + optional at the store. +- [`habits-app.md`](./habits-app.md) still defines an ontology. That is + the intended external-app shape. + +## Checklist + +- [x] Decision: schema recommended, not required. +- [x] Rust `Resource::set` accepts unknown Property URLs. +- [x] Unit test: classless resource + unknown property saves and reloads. +- [x] Unit test: known Property still rejects a datatype mismatch. +- [x] FAQ + JS getting-started mention the classless path. +- [ ] Soften JSON-AD parse to infer JSON types when the Property is + missing (only if an importer hits it). +- [ ] `defineSchema` / `registerSchema` (tracked in + [`json-schema-code-first.md`](./json-schema-code-first.md)). +- [ ] Tutorial leads with code-first schema, shows one schemaless + snippet.