diff --git a/.env.example b/.env.example index 88c0212..941c7ff 100644 --- a/.env.example +++ b/.env.example @@ -27,7 +27,9 @@ API_CONSTANTS=owner=localthought,repo=test-repo-1 STORE_DIR=data/store # macOS: export STORE_DIR="$HOME/Library/Application Support/atomic-data/store" -# Your AtomicServer agent subject; grants access to newly created drives only. +# Your AtomicServer agent subject; grants access to the imported drive and, +# if that agent already has a local resource, adds the drive to its +# saved-drive list so it shows up in the app. # DRIVE_OWNER=did:ad:agent:YOUR_PUBLIC_KEY # Interactive OAuth fallback, used when API_TOKEN is missing or GitHub no diff --git a/CLAUDE.md b/CLAUDE.md index 5b8a279..aa0f9d1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -56,7 +56,7 @@ in `main.rs`, not in a `pub mod` under `src/lib.rs`. | `src/oauth.rs` | Interactive GitHub OAuth fallback: validates a configured PAT against `GET /user`, and if that fails or no PAT is configured, runs a local web server through the authorization-code flow. Only active when `OAUTH_CLIENT_ID`/`OAUTH_CLIENT_SECRET` are set. `#[cfg(not(target_arch = "wasm32"))]` — binds a real TCP listener, which has no wasm32 equivalent. | | `src/http.rs` | `ReqwestFetch`, the sync engine's one HTTP extension point (`syncables::client::client::Fetch`). Its impl is `?Send` on wasm32 (see [WASM compatibility](README.md#wasm-compatibility)). | | `src/ontology.rs` | `SubjectMapper` — the `internal:/…` ⇄ `/…` subject mapping that keeps minted ontology terms resolvable. | -| `src/store.rs` | `AtomicStorage` — renders `syncables-rs`'s plain JSON records and neutral ontology description into Atomic Data resources in a `Storelike`. Its `Storage` impl is `?Send` on wasm32, same reason as `http.rs`. | +| `src/store.rs` | `AtomicStorage` — renders `syncables-rs`'s plain JSON records and neutral ontology description into Atomic Data resources in a `Storelike`, grouped under one Drive/DocumentV2/Table per imported dataset (`with_dataset`). Its `Storage` impl is `?Send` on wasm32, same reason as `http.rs`. | | `src/main.rs` | Wires configuration, credential resolution, the store, and the sync engine together, then runs one sync and exports it as JSON-AD. | ## Conventions to preserve diff --git a/README.md b/README.md index ee39322..a067506 100644 --- a/README.md +++ b/README.md @@ -140,6 +140,9 @@ engine's dependency tree. | Class | `internal:/github-issues/class/issue` | `/github-issues/class/issue` | | Property | `internal:/github-issues/property/title` | `/github-issues/property/title` | | Record | `internal:/localthought%2Ftest-repo-1/issue/1` | `/localthought%2Ftest-repo-1/issue/1` | +| Drive | `internal:/reflector-drives/localthought%2Ftest-repo-1` | `/reflector-drives/localthought%2Ftest-repo-1` | +| Document | `internal:/reflector-drives/localthought%2Ftest-repo-1/document` | `/reflector-drives/localthought%2Ftest-repo-1/document` | +| Table | `internal:/reflector-drives/localthought%2Ftest-repo-1/table/issue` | `/reflector-drives/localthought%2Ftest-repo-1/table/issue` | Records are `internal:///`, each segment escaped so a namespace containing `/` stays a single segment. Fields are typed with the @@ -154,14 +157,47 @@ and defaults to `data/store` (creating `data/store/atomic.redb`). Old `data/store.json-ad` exports are not imported automatically: run a full sync to populate the database. -Each namespace gets a separate Drive at -`internal:/reflector-drives/`, named from the ontology and -namespace: for example, **github issues localthought test-repo-1**. Issues, -comments, and their nested resources have `parent` and `drive` pointing to -that drive. The user's main drive is not changed. Ontology terms keep their -shared canonical paths shown above. Repeated syncs reuse the repository drive -and retain its permissions; updating resources builds on their stored Loro -state. +### The imported drive, document and table + +Each **imported dataset** — the complete set of records one `cargo run` +syncs, named by `API_CONSTANTS`' values (`owner=localthought,repo=test-repo-1` +→ `localthought/test-repo-1`) — gets *one* Drive at +`internal:/reflector-drives/`, named from the ontology and +dataset: for example, **github issues localthought test-repo-1**. This is one +Drive per dataset regardless of how many nested collections it has: issue +comments are addressed under a deeper, per-issue namespace +(`localthought/test-repo-1/1`, one per issue) so they still dedupe correctly, +but they file into this same Drive rather than one of their own — a document's +complete imported dataset, comments included, never splits across drives. + +Under that Drive sits one **DocumentV2** (`.../document`), and under the +document, one native AD **Table** per root-level resource the document +declares (just `.../table/issue` for the vendored GitHub Issues document). +A table's `classtype` is the resource's own ontology Class, so every property +that Class recommends or requires — for `issue`, at least number, title, +state, body, author (`user`), updated time (`updated_at`) and source URL +(`html_url`) — renders as a column. Every `issue` record's `parent` is that +table (so it shows up as a row); comments and other nested records keep +`parent` pointing straight at the Drive. Every resource's `drive` property +(used for permission/fan-out scoping, independent of `parent`) always points +at the Drive itself. + +The user's main drive is not changed. Ontology terms keep their shared +canonical paths shown above. Repeated syncs reuse the repository Drive, +Document and Table(s) and retain their permissions; updating resources builds +on their stored Loro state. If `DRIVE_OWNER` is set, the Drive is also added +to that agent's saved-drive list (idempotently) on every sync, so it's +discoverable in the app without a manual share — this only updates an +already-locally-stored Agent resource, never fetches or invents one. + +**Upgrading from before this existed:** an older reflector-rs (before +[issue #18](https://github.com/localthought/reflector-rs/issues/18)) keyed a +Drive off each record's own namespace, so every issue's comments fragmented +into their own separate Drive instead of sharing the dataset's one Drive. The +next sync against an existing store migrates automatically: the one Drive +matching the dataset itself is reused as-is, every comment gets re-pointed at +it as it's re-synced (routine — every sync re-puts every record), and the +leftover per-issue Drives are removed. ### Sharing AtomicServer's database on macOS @@ -191,11 +227,15 @@ is not expanded by Reflector. The binary reads process environment variables; it does not load `.env` automatically. Set `DRIVE_OWNER` before the first sync so your AtomicServer agent can read and -edit the new drive. Without it, no user read/write grants are added (server +edit the new drive — the same sync also adds it to that agent's saved-drive +list, so it shows up in the app's drive switcher without a manual share. +Without `DRIVE_OWNER`, no user read/write grants are added (server administration can still access it). Reflector never makes imported private issues public by default. To change permissions on an existing drive, use AtomicServer's sharing controls. Open the drive directly at -`/reflector-drives/localthought%2Ftest-repo-1`. +`/reflector-drives/localthought%2Ftest-repo-1`, or its document at +`/reflector-drives/localthought%2Ftest-repo-1/document` for the +issues table. ## The vendored document diff --git a/docs/ai-logs/sessions/2026-09-07-issue-18-drive-document-table.md b/docs/ai-logs/sessions/2026-09-07-issue-18-drive-document-table.md new file mode 100644 index 0000000..41c22ab --- /dev/null +++ b/docs/ai-logs/sessions/2026-09-07-issue-18-drive-document-table.md @@ -0,0 +1,87 @@ +# Issue 18: user-owned drive with a document and issues table + +Assistant: Claude Code. + +## Human request + +Work on https://github.com/localthought/reflector-rs/issues/18, update docs +as the work progresses, and merge at will. + +The issue asks for one discoverable Atomic Data drive per imported +repository/dataset (not one per record namespace, which today fragments a +repository's issue comments into a separate drive per issue), registered +through the normal saved-drive mechanism; a DocumentV2 under that drive with +a native AD table of the imported issues (number, title, state, body, +author, updated time, source URL); comments and nested data staying +accessible in that same drive; consistent `parent`/`drive` properties; +reuse/idempotence across reimports and restarts; and a safe transition for +drives created by the earlier per-namespace scheme. + +## Investigation + +Read `store.rs`'s `AtomicStorage` and `Storage::put`, `config.rs`, and +`main.rs` to see how drives are currently created (keyed off each record's +own `namespace`, which for a nested collection like issue comments is +`owner/repo/` — one drive per issue). Confirmed against +`syncables-rs`'s `sync::client::walk_all` that a nested collection's +namespace is always a deeper extension of its dataset's own root namespace, +and that root-level records are always synced (and `put`) before anything +nested under them. + +Cloned `ontola/atomic-server` at the pinned revision to check the actual +resource shapes: `urls::TABLE`'s `classtype` names a row Class (confirmed +against `browser/data-browser/src/chunks/TablePage/createTableFromSpec.ts` +and its AI skill doc `creating-tables.md`), a Table's rows are queried by +`parent == table AND is-a == classtype` (`useTableData.ts`), permission +inheritance climbs the `parent` chain while `drive` is a separate direct +pointer used for fan-out (`lib/src/hierarchy.rs`, `resources.rs::get_drive`), +and the saved-drive list is the Agent's `drives` property (`lib/src/db.rs`'s +private `push_drive_to_list`, generalized here against `Storelike` since +`AtomicStorage` isn't always a `Db`). `DocumentV2`'s real collaborative body +is a loro-prosemirror CRDT tree with no Rust-side construction path in this +codebase or a vendored reference implementation; `documentContent` as an +ordinary Markdown property (the format `extract_document_plain_text`'s own +fallback reads) was used instead of attempting to hand-build that tree. + +## Changes and results + +- `AtomicStorage` now takes a `dataset` (`with_dataset`, e.g. + `owner/repo`), computed by a new `Config::dataset_namespace` from + `API_CONSTANTS`' values in key order — one Drive per dataset, independent + of any individual record's own (possibly deeper) namespace. +- Added one DocumentV2 and one native AD Table per root-level resource + (`ensure_document`/`ensure_table`), both idempotent and created lazily on + the first matching `put`. A root-level record's `parent` is now its + table (so it renders as a row); nested records (comments) keep `parent` + pointing at the drive, same drive as the root records. `drive` always + points at the drive itself either way. +- `ensure_drive` now also registers the drive on `DRIVE_OWNER`'s `drives` + property when that agent already has a local resource, and removes any + drive left over from the old per-namespace scheme under this dataset + (`migrate_legacy_drives`) — the records that pointed at a removed drive + are re-pointed to the real one automatically the next time they're + `put`, since `persist` always rewrites `parent`/`drive` from scratch. +- Added `html_url` to the vendored `issue` schema (`spec/`) as the issue's + source URL, so it flows through the existing ontology derivation as a + table column with no `syncables-rs` changes. +- Updated/added `store.rs` unit tests for containment (record → table → + document → drive), nested collections staying off the table, saved-drive + registration idempotence, and legacy-drive migration; rewrote the + restart test's "unrelated repository" case as two dataset-scoped + `AtomicStorage` instances sharing one `Db`, matching how two real syncs + would actually share an AtomicServer database. Updated `README.md` and + `CLAUDE.md`. +- `cargo fmt --check`, `cargo test` (43 tests), `cargo clippy --all-targets + -- -D warnings`, `cargo build`, and both `--target wasm32-unknown-unknown` + checks all pass locally. + +Not covered: this only verifies the resource shapes atomic-server's own +Rust code and TypeScript client documented, not an actual rendered document +open in the data browser — there is no browser or running atomic-server +frontend in this environment to click through. `documentContent` is set as +a Markdown summary rather than the browser's own collaborative-editor +representation, for the reason above; the Table's `classtype` reuses the +existing per-resource ontology Class as-is (so the table also gets +`id`/`state_reason`/`labels` columns beyond the seven the issue names) +rather than hand-building a second, narrower row Class duplicating those +same property subjects. Both are called out in the PR description. diff --git a/spec/github-issues.openapi.yaml b/spec/github-issues.openapi.yaml index a867645..ab61f05 100644 --- a/spec/github-issues.openapi.yaml +++ b/spec/github-issues.openapi.yaml @@ -47,6 +47,10 @@ components: updated_at: type: string format: date-time + html_url: + type: string + format: uri + description: The issue's page on github.com — its source URL. user: $ref: '#/components/schemas/user' labels: diff --git a/src/config.rs b/src/config.rs index 2e7e091..d8aa215 100644 --- a/src/config.rs +++ b/src/config.rs @@ -261,6 +261,21 @@ impl Config { }) } + /// Identifies the complete dataset this run imports — e.g. + /// `localthought/test-repo-1` — derived from [`Config::constants`]' + /// values in key order (`owner` then `repo`, for the default document). + /// Every record the sync writes, root or nested, is grouped under the + /// *one* Drive/Document/Table this names — see + /// [`crate::store::AtomicStorage::with_dataset`] — rather than each + /// record's own (possibly deeper, per-parent) namespace. + pub fn dataset_namespace(&self) -> String { + self.constants + .values() + .cloned() + .collect::>() + .join("/") + } + /// Fails early on anything the sync would only discover mid-flight: a /// missing document or overlay is a deployment mistake, not a sync error. pub fn validate(&self) -> Result<()> { @@ -355,6 +370,16 @@ mod tests { assert_eq!(constants["repo"], "test-repo-1"); } + #[test] + fn dataset_namespace_joins_constant_values_in_key_order() { + let config = Config::from_lookup(Path::new("/reflector"), |key| match key { + env_var::PUBLIC_URL => Some("http://localhost:9883".into()), + _ => None, + }) + .unwrap(); + assert_eq!(config.dataset_namespace(), "localthought/test-repo-1"); + } + #[test] fn public_url_loses_its_trailing_slash() { assert_eq!( diff --git a/src/main.rs b/src/main.rs index 7c5866d..f35e46b 100644 --- a/src/main.rs +++ b/src/main.rs @@ -68,7 +68,8 @@ async fn main() -> Result<()> { Arc::new(store), SubjectMapper::new(config.public_url.clone()), ) - .with_drive_owner(config.drive_owner.clone()); + .with_drive_owner(config.drive_owner.clone()) + .with_dataset(config.dataset_namespace()); let client = SyncClient::new( ClientConfig { diff --git a/src/store.rs b/src/store.rs index d41e1aa..39a3cca 100644 --- a/src/store.rs +++ b/src/store.rs @@ -69,6 +69,15 @@ pub struct AtomicStorage { /// than an `&mut self` because [`Storage`] hands out shared references. index: RwLock, drive_owner: Option, + /// Identifies the complete imported dataset this instance writes into — + /// e.g. `localthought/test-repo-1` — set via [`Self::with_dataset`]. + /// Every record this instance ever `put`s, root or nested, is grouped + /// under the *one* Drive (and Document/Table) this names, regardless of + /// which namespace an individual record's own collection carries: a + /// nested collection (issue comments) has a deeper namespace so its + /// records still address and dedupe correctly, but it belongs to this + /// same dataset, not a namespace of its own. + dataset: String, } impl AtomicStorage { @@ -78,6 +87,7 @@ impl AtomicStorage { mapper, index: RwLock::new(TermIndex::default()), drive_owner: None, + dataset: String::new(), } } @@ -87,13 +97,40 @@ impl AtomicStorage { self } - pub fn drive_subject(&self, namespace: &str) -> String { - self.mapper - .internal(&format!("reflector-drives/{}", encode_segment(namespace))) + /// Names the imported dataset (e.g. `owner/repo`) every record this + /// instance writes is grouped under. + pub fn with_dataset(mut self, dataset: impl Into) -> Self { + self.dataset = dataset.into(); + self + } + + /// `internal:/reflector-drives/` — the one Drive this instance's + /// records, Document and Table(s) all live under. + pub fn drive_subject(&self) -> String { + self.mapper.internal(&format!( + "reflector-drives/{}", + encode_segment(&self.dataset) + )) + } + + /// `internal:/reflector-drives//document` — the one DocumentV2 + /// this dataset's table(s) are filed under. + fn document_subject(&self) -> String { + format!("{}/document", self.drive_subject()) } - async fn ensure_drive(&self, namespace: &str) -> Result { - let subject = Subject::from(self.drive_subject(namespace).as_str()); + /// `internal:/reflector-drives//table/` — the native + /// AD table for one root-level resource of this dataset (e.g. `issue`). + fn table_subject(&self, resource: &str) -> String { + format!( + "{}/table/{}", + self.drive_subject(), + encode_segment(resource) + ) + } + + async fn ensure_drive(&self) -> Result { + let subject = Subject::from(self.drive_subject().as_str()); if !self.store.has_stored_resource(&subject) { let mut drive = Resource::new(subject.to_string()); set!( @@ -110,7 +147,7 @@ impl AtomicStorage { set!( drive, urls::NAME.to_owned(), - Value::String(format!("{} {}", label, namespace.replace('/', " "))) + Value::String(format!("{} {}", label, self.dataset.replace('/', " "))) ); if let Some(owner) = &self.drive_owner { for permission in [urls::READ, urls::WRITE] { @@ -128,6 +165,188 @@ impl AtomicStorage { .await .map_err(|error| StorageError::new(error.to_string()))?; } + if let Some(owner) = &self.drive_owner { + self.register_saved_drive(owner, &subject).await?; + } + self.migrate_legacy_drives(&subject).await?; + Ok(subject) + } + + /// Adds `drive` to `owner`'s saved-drive list — the `drives` property the + /// Atomic Data browser reads to show a drive as one of "your" drives — + /// mirroring `atomic_lib`'s own (private, `Db`-specific) + /// `push_drive_to_list`, generalized to any [`Storelike`] since this + /// store isn't always a `Db`. Idempotent (an already-listed drive isn't + /// duplicated) and run on every call, not just when the drive is first + /// created, so a drive from before this existed still gets registered. + /// A no-op when `owner` has no local resource yet: appending to one we'd + /// have to fetch over the network would fight this store's local-first + /// design, and inventing an Agent resource here isn't this store's job. + async fn register_saved_drive(&self, owner: &str, drive: &Subject) -> Result<(), StorageError> { + let owner_subject = Subject::from(owner); + if !self.store.has_stored_resource(&owner_subject) { + return Ok(()); + } + let mut agent = self + .store + .get_resource(&owner_subject) + .await + .map_err(|error| StorageError::new(error.to_string()))?; + let mut drives: Vec = match agent.get(urls::DRIVES) { + Ok(Value::ResourceArray(items)) => items.clone(), + _ => Vec::new(), + }; + if drives.iter().any(|item| item.to_string() == drive.as_str()) { + return Ok(()); + } + drives.push(SubResource::Subject(drive.clone())); + set!(agent, urls::DRIVES.to_owned(), Value::ResourceArray(drives)); + self.store + .add_resource_opts(&agent, false, true, true) + .await + .map_err(|error| StorageError::new(error.to_string())) + } + + /// Removes a Drive left over from before a Drive was keyed off the whole + /// dataset rather than each record's own namespace: a nested collection + /// (issue comments, one namespace per parent issue) used to fragment + /// into a separate Drive per parent instead of sharing this dataset's + /// one Drive. The records that used to point at a removed Drive are + /// re-pointed to `current` the next time they're `put` — `persist` + /// always rewrites `parent`/`drive` from the freshly computed value, so + /// nothing else has to migrate them — this only cleans up the + /// now-unreferenced Drive resources themselves. + async fn migrate_legacy_drives(&self, current: &Subject) -> Result<(), StorageError> { + if self.dataset.is_empty() { + return Ok(()); + } + let prefix = self.mapper.internal(&format!( + "reflector-drives/{}%2F", + encode_segment(&self.dataset) + )); + let stale: Vec = self + .store + .all_resources(false) + .filter_map(|resource| { + let subject = resource.get_subject().to_string(); + let is_stale_drive = subject.starts_with(&prefix) + && subject.as_str() != current.as_str() + && resource + .get(urls::IS_A) + .is_ok_and(|is_a| is_a.to_string().contains(urls::DRIVE)); + is_stale_drive.then(|| Subject::from(subject.as_str())) + }) + .collect(); + for subject in stale { + self.store + .remove_resource(&subject) + .await + .map_err(|error| StorageError::new(error.to_string()))?; + } + Ok(()) + } + + /// Ensures the one native AD table for a root-level resource (e.g. + /// `issue`) exists under this dataset's Document, and returns it — every + /// record of that resource is filed under it as `parent`, so it shows up + /// as a row. Falls back to `drive` for a resource this dataset's + /// ontology has no Class for, which shouldn't happen (the engine derives + /// one per `crudResources` entry) but must not fail the sync if it does. + async fn ensure_table( + &self, + drive: &Subject, + resource: &str, + index: &TermIndex, + ) -> Result { + let Some(class) = index.classes.get(&ontology_shortname(resource)) else { + return Ok(drive.clone()); + }; + let document = self.ensure_document(drive, index).await?; + let subject = Subject::from(self.table_subject(resource).as_str()); + if !self.store.has_stored_resource(&subject) { + let mut table = Resource::new(subject.to_string()); + set!( + table, + urls::IS_A.to_owned(), + Value::ResourceArray(vec![SubResource::Subject(Subject::from(urls::TABLE))]) + ); + set!( + table, + urls::NAME.to_owned(), + Value::String(table_name(resource)) + ); + set!( + table, + urls::CLASSTYPE_PROP.to_owned(), + Value::AtomicUrl(Subject::from(class.as_str())) + ); + set!( + table, + urls::PARENT.to_owned(), + Value::AtomicUrl(document.clone()) + ); + set!( + table, + urls::DRIVE_PROP.to_owned(), + Value::AtomicUrl(drive.clone()) + ); + self.store + .add_resource(&table) + .await + .map_err(|error| StorageError::new(error.to_string()))?; + } + Ok(subject) + } + + /// Ensures the one DocumentV2 this dataset's table(s) are filed under. + async fn ensure_document( + &self, + drive: &Subject, + index: &TermIndex, + ) -> Result { + let subject = Subject::from(self.document_subject().as_str()); + if !self.store.has_stored_resource(&subject) { + let label = if index.ontology_path.is_empty() { + "reflected data".to_owned() + } else { + index.ontology_path.replace('-', " ") + }; + let mut document = Resource::new(subject.to_string()); + set!( + document, + urls::IS_A.to_owned(), + Value::ResourceArray(vec![SubResource::Subject(Subject::from(urls::DOCUMENT_V2))]) + ); + set!( + document, + urls::NAME.to_owned(), + Value::String(format!("{label} {}", self.dataset)) + ); + set!( + document, + urls::DOCUMENT_CONTENT.to_owned(), + Value::Markdown(format!( + "# {label} {}\n\nImported by reflector-rs. The table(s) filed under \ + this document list the synced records; comments and other nested \ + data are kept in this same drive.\n", + self.dataset + )) + ); + set!( + document, + urls::PARENT.to_owned(), + Value::AtomicUrl(drive.clone()) + ); + set!( + document, + urls::DRIVE_PROP.to_owned(), + Value::AtomicUrl(drive.clone()) + ); + self.store + .add_resource(&document) + .await + .map_err(|error| StorageError::new(error.to_string()))?; + } Ok(subject) } @@ -496,7 +715,18 @@ impl Storage for AtomicStorage { async fn put(&self, record: &Record) -> Result<(), StorageError> { let index = self.index(); let subject = self.record_subject(&record.namespace, &record.resource, &record.id); - let drive = self.ensure_drive(&record.namespace).await?; + let drive = self.ensure_drive().await?; + // A root-level record's namespace is exactly this dataset (no + // parent-provided context, e.g. `owner/repo`); a nested collection's + // namespace is deeper (e.g. `owner/repo/1` for one issue's + // comments). Only root-level records are filed under this + // resource's table — comments and other nested data stay filed + // directly under the drive, in the same drive either way. + let parent = if record.namespace == self.dataset { + self.ensure_table(&drive, &record.resource, &index).await? + } else { + drive.clone() + }; let stale_prefix = format!("{subject}/"); let stale_subjects: Vec = self .store @@ -525,7 +755,7 @@ impl Storage for AtomicStorage { set!( resource, urls::PARENT.to_owned(), - Value::AtomicUrl(drive.clone()) + Value::AtomicUrl(parent.clone()) ); set!( resource, @@ -682,6 +912,21 @@ impl Storage for AtomicStorage { } } +/// A table's display name from its resource name: `issue` → `Issues`. A +/// blunt English pluralization (an `s` suffix), good enough for the plain +/// resource names `crudResources` declares without pulling in a real +/// pluralization dependency for this cosmetic a purpose. +fn table_name(resource: &str) -> String { + let mut name = String::with_capacity(resource.len() + 1); + let mut chars = resource.chars(); + if let Some(first) = chars.next() { + name.extend(first.to_uppercase()); + } + name.push_str(chars.as_str()); + name.push('s'); + name +} + /// JSON → Atomic Data. The ontology's datatype decides when it can (a string /// the document typed as a timestamp is stored as one); otherwise the JSON /// type does. @@ -895,6 +1140,7 @@ mod tests { Arc::new(store), SubjectMapper::new("https://my-ontologies.com"), ) + .with_dataset("localthought/test-repo-1") } #[tokio::test] @@ -904,6 +1150,11 @@ mod tests { issue .value .insert("user".into(), serde_json::json!({"login": "alice"})); + // A second dataset, synced by a second `AtomicStorage` (its own + // `dataset`) sharing the same underlying `Db` — the realistic + // shape of "unrelated data already in this AtomicServer database", + // since one reflector-rs run's `AtomicStorage` only ever writes one + // dataset (see `README.md`'s "Sharing AtomicServer's database"). let mut other = issue.clone(); other.namespace = "another/repository".into(); let owner = "did:ad:agent:test-owner"; @@ -921,14 +1172,18 @@ mod tests { .set_unsafe(urls::NAME.into(), Value::String("My main drive".into())) .unwrap(); db.add_resource(&main_drive).await.unwrap(); - let storage = AtomicStorage::new( - Arc::new(db), - SubjectMapper::new("https://my-ontologies.com"), - ) - .with_drive_owner(Some(owner.into())); + let db = Arc::new(db); + let storage = + AtomicStorage::new(db.clone(), SubjectMapper::new("https://my-ontologies.com")) + .with_drive_owner(Some(owner.into())) + .with_dataset("localthought/test-repo-1"); storage.put_ontology(&ontology()).await.unwrap(); storage.put(&issue).await.unwrap(); - storage.put(&other).await.unwrap(); + let other_storage = + AtomicStorage::new(db, SubjectMapper::new("https://my-ontologies.com")) + .with_dataset("another/repository"); + other_storage.put_ontology(&ontology()).await.unwrap(); + other_storage.put(&other).await.unwrap(); let subject = Subject::from( storage .record_subject(&issue.namespace, &issue.resource, &issue.id) @@ -955,7 +1210,8 @@ mod tests { let storage = AtomicStorage::new( Arc::new(db), SubjectMapper::new("https://my-ontologies.com"), - ); + ) + .with_dataset("localthought/test-repo-1"); storage.put_ontology(&ontology()).await.unwrap(); assert_eq!( storage @@ -966,7 +1222,7 @@ mod tests { .value["user"], issue.value["user"] ); - let drive = Subject::from(storage.drive_subject(&issue.namespace).as_str()); + let drive = Subject::from(storage.drive_subject().as_str()); let stored_drive = storage.store.get_resource(&drive).await.unwrap(); assert!(stored_drive.get_propvals()[urls::WRITE] .to_string() @@ -1032,33 +1288,38 @@ mod tests { let storage = AtomicStorage::new( Arc::new(db), SubjectMapper::new("https://my-ontologies.com"), - ); + ) + .with_dataset("localthought/test-repo-1"); storage.put_ontology(&ontology()).await.unwrap(); assert!(storage .get(&issue.namespace, &issue.resource, &issue.id) .await .unwrap() .is_none()); + let other_storage = AtomicStorage::new( + storage.store.clone(), + SubjectMapper::new("https://my-ontologies.com"), + ) + .with_dataset("another/repository"); assert_eq!( - storage + other_storage .list(&other.namespace, &other.resource) .await .unwrap() .len(), 1 ); - assert_ne!( - storage.drive_subject(&issue.namespace), - storage.drive_subject(&other.namespace) - ); + assert_ne!(storage.drive_subject(), other_storage.drive_subject()); } #[tokio::test] - async fn records_belong_to_a_repository_drive() { + async fn root_level_records_are_filed_under_a_table_under_the_dataset_document() { let storage = storage().await; storage.put_ontology(&ontology()).await.unwrap(); let record = record(); storage.put(&record).await.unwrap(); + let drive = Subject::from(storage.drive_subject().as_str()); + let stored = storage .store .get_resource(&Subject::from( @@ -1068,21 +1329,63 @@ mod tests { )) .await .unwrap(); - let parent = stored - .get_propvals() - .get(urls::PARENT) - .expect("record needs a drive") - .to_string(); - let drive = storage + // `drive` is a direct pointer to the drive root (used for ACL/fan-out + // scoping); `parent` is the actual containing resource — the table, + // not the drive — which is what makes the record show up as a row. + assert_eq!( + stored.get_propvals()[urls::DRIVE_PROP].to_string(), + drive.to_string() + ); + let table_subject = stored.get_propvals()[urls::PARENT].to_string(); + assert_ne!(table_subject, drive.to_string()); + + let table = storage .store - .get_resource(&Subject::from(parent.as_str())) + .get_resource(&Subject::from(table_subject.as_str())) .await - .unwrap(); + .expect("table"); + assert!(table + .get(urls::IS_A) + .unwrap() + .to_string() + .contains(urls::TABLE)); assert_eq!( - drive.get_propvals()[urls::NAME].to_string(), + table.get(urls::CLASSTYPE_PROP).unwrap().to_string(), + "internal:/github-issues/class/issue" + ); + assert_eq!(table.get(urls::NAME).unwrap().to_string(), "Issues"); + assert_eq!( + table.get(urls::DRIVE_PROP).unwrap().to_string(), + drive.to_string() + ); + + let document_subject = table.get(urls::PARENT).unwrap().to_string(); + let document = storage + .store + .get_resource(&Subject::from(document_subject.as_str())) + .await + .expect("document"); + assert!(document + .get(urls::IS_A) + .unwrap() + .to_string() + .contains(urls::DOCUMENT_V2)); + assert!(document.get(urls::DOCUMENT_CONTENT).is_ok()); + assert_eq!( + document.get(urls::PARENT).unwrap().to_string(), + drive.to_string() + ); + assert_eq!( + document.get(urls::DRIVE_PROP).unwrap().to_string(), + drive.to_string() + ); + + let stored_drive = storage.store.get_resource(&drive).await.unwrap(); + assert_eq!( + stored_drive.get_propvals()[urls::NAME].to_string(), "github issues localthought test-repo-1" ); - assert_eq!(stored.get_propvals()[urls::DRIVE_PROP].to_string(), parent); + let roundtrip = storage .get(&record.namespace, &record.resource, &record.id) .await @@ -1092,6 +1395,125 @@ mod tests { assert!(!roundtrip.value.contains_key("drive")); } + #[tokio::test] + async fn nested_collections_stay_directly_under_the_drive_not_the_table() { + let storage = storage().await; + storage.put_ontology(&ontology()).await.unwrap(); + storage.put(&record()).await.unwrap(); + let comment = Record { + namespace: "localthought/test-repo-1/1".to_owned(), + resource: "issueComment".to_owned(), + id: "5449492104".to_owned(), + value: serde_json::json!({ "body": "hi" }) + .as_object() + .unwrap() + .clone(), + }; + storage.put(&comment).await.unwrap(); + + let stored = storage + .store + .get_resource(&Subject::from( + storage + .record_subject(&comment.namespace, &comment.resource, &comment.id) + .as_str(), + )) + .await + .unwrap(); + let drive = Subject::from(storage.drive_subject().as_str()); + // Comments are nested under a per-issue namespace ("owner/repo/1"), + // not the dataset's own root namespace — they file straight under + // the drive rather than the issues table, but the *same* drive as + // the root-level issue records (this is the issue #18 fix: no + // separate drive per comment namespace). + assert_eq!( + stored.get_propvals()[urls::PARENT].to_string(), + drive.to_string() + ); + assert_eq!( + stored.get_propvals()[urls::DRIVE_PROP].to_string(), + drive.to_string() + ); + } + + #[tokio::test] + async fn a_pre_existing_agent_is_registered_as_a_saved_drive_idempotently() { + let storage = storage().await; + let owner = "https://example.com/agents/alice"; + let mut agent = Resource::new(owner.to_owned()); + agent + .set_unsafe(urls::NAME.into(), Value::String("Alice".into())) + .unwrap(); + storage.store.add_resource(&agent).await.unwrap(); + let storage = storage.with_drive_owner(Some(owner.to_owned())); + + storage.put_ontology(&ontology()).await.unwrap(); + storage.put(&record()).await.unwrap(); + // A second sync (the common case: a periodic re-sync) must not + // duplicate the entry. + storage.put(&record()).await.unwrap(); + + let stored_agent = storage + .store + .get_resource(&Subject::from(owner)) + .await + .unwrap(); + let drives = stored_agent.get(urls::DRIVES).unwrap().to_string(); + let drive = storage.drive_subject(); + let occurrences = drives.matches(drive.as_str()).count(); + assert_eq!(occurrences, 1, "{drives}"); + } + + #[tokio::test] + async fn a_legacy_per_namespace_drive_is_migrated_away() { + let storage = storage().await; + storage.put_ontology(&ontology()).await.unwrap(); + + // Simulate a Drive created by the pre-issue-18 code, keyed off a + // nested collection's own namespace ("owner/repo/1" for one issue's + // comments) rather than the whole dataset. + let legacy_subject = "internal:/reflector-drives/localthought%2Ftest-repo-1%2F1"; + let mut legacy_drive = Resource::new(legacy_subject.to_owned()); + legacy_drive + .set_unsafe( + urls::IS_A.into(), + Value::ResourceArray(vec![SubResource::Subject(Subject::from(urls::DRIVE))]), + ) + .unwrap(); + storage.store.add_resource(&legacy_drive).await.unwrap(); + + // Any `put` (comments included) re-ensures the drive and migrates. + let comment = Record { + namespace: "localthought/test-repo-1/1".to_owned(), + resource: "issueComment".to_owned(), + id: "5449492104".to_owned(), + value: serde_json::json!({ "body": "hi" }) + .as_object() + .unwrap() + .clone(), + }; + storage.put(&comment).await.unwrap(); + + assert!(!storage + .store + .has_stored_resource(&Subject::from(legacy_subject))); + // The now-migrated comment is re-pointed to the real drive by its + // own fresh `put`, unaffected by the legacy drive's removal. + let stored_comment = storage + .store + .get_resource(&Subject::from( + storage + .record_subject(&comment.namespace, &comment.resource, &comment.id) + .as_str(), + )) + .await + .unwrap(); + assert_eq!( + stored_comment.get_propvals()[urls::DRIVE_PROP].to_string(), + storage.drive_subject() + ); + } + #[tokio::test] async fn ontology_terms_are_stored_at_their_public_paths() { let storage = storage().await;