Group imports into one drive with a document and native issues table

.env.example

@@ -27,7 +27,9 @@ API_CONSTANTS=owner=localthought,repo=test-repo-1
 STORE_DIR=data/store
 # macOS: export STORE_DIR="$HOME/Library/Application Support/atomic-data/store"
 
-# Your AtomicServer agent subject; grants access to newly created drives only.
+# Your AtomicServer agent subject; grants access to the imported drive and,
+# if that agent already has a local resource, adds the drive to its
+# saved-drive list so it shows up in the app.
 # DRIVE_OWNER=did:ad:agent:YOUR_PUBLIC_KEY
 
 # Interactive OAuth fallback, used when API_TOKEN is missing or GitHub no

CLAUDE.md

@@ -56,7 +56,7 @@ in `main.rs`, not in a `pub mod` under `src/lib.rs`.
 | `src/oauth.rs` | Interactive GitHub OAuth fallback: validates a configured PAT against `GET /user`, and if that fails or no PAT is configured, runs a local web server through the authorization-code flow. Only active when `OAUTH_CLIENT_ID`/`OAUTH_CLIENT_SECRET` are set. `#[cfg(not(target_arch = "wasm32"))]` — binds a real TCP listener, which has no wasm32 equivalent. |
 | `src/http.rs` | `ReqwestFetch`, the sync engine's one HTTP extension point (`syncables::client::client::Fetch`). Its impl is `?Send` on wasm32 (see [WASM compatibility](README.md#wasm-compatibility)). |
 | `src/ontology.rs` | `SubjectMapper` — the `internal:/…` ⇄ `<PUBLIC_URL>/…` subject mapping that keeps minted ontology terms resolvable. |
-| `src/store.rs` | `AtomicStorage` — renders `syncables-rs`'s plain JSON records and neutral ontology description into Atomic Data resources in a `Storelike`. Its `Storage` impl is `?Send` on wasm32, same reason as `http.rs`. |
+| `src/store.rs` | `AtomicStorage` — renders `syncables-rs`'s plain JSON records and neutral ontology description into Atomic Data resources in a `Storelike`, grouped under one Drive/DocumentV2/Table per imported dataset (`with_dataset`). Its `Storage` impl is `?Send` on wasm32, same reason as `http.rs`. |
 | `src/main.rs` | Wires configuration, credential resolution, the store, and the sync engine together, then runs one sync and exports it as JSON-AD. |
 
 ## Conventions to preserve

README.md

@@ -140,6 +140,9 @@ engine's dependency tree.
 | Class | `internal:/github-issues/class/issue` | `<PUBLIC_URL>/github-issues/class/issue` |
 | Property | `internal:/github-issues/property/title` | `<PUBLIC_URL>/github-issues/property/title` |
 | Record | `internal:/localthought%2Ftest-repo-1/issue/1` | `<PUBLIC_URL>/localthought%2Ftest-repo-1/issue/1` |
+| Drive | `internal:/reflector-drives/localthought%2Ftest-repo-1` | `<PUBLIC_URL>/reflector-drives/localthought%2Ftest-repo-1` |
+| Document | `internal:/reflector-drives/localthought%2Ftest-repo-1/document` | `<PUBLIC_URL>/reflector-drives/localthought%2Ftest-repo-1/document` |
+| Table | `internal:/reflector-drives/localthought%2Ftest-repo-1/table/issue` | `<PUBLIC_URL>/reflector-drives/localthought%2Ftest-repo-1/table/issue` |
 
 Records are `internal:/<namespace>/<resource>/<id>`, each segment escaped so a
 namespace containing `/` stays a single segment. Fields are typed with the
@@ -154,14 +157,47 @@ and defaults to `data/store` (creating `data/store/atomic.redb`). Old
 `data/store.json-ad` exports are not imported automatically: run a full sync
 to populate the database.
 
-Each namespace gets a separate Drive at
-`internal:/reflector-drives/<escaped-namespace>`, named from the ontology and
-namespace: for example, **github issues localthought test-repo-1**. Issues,
-comments, and their nested resources have `parent` and `drive` pointing to
-that drive. The user's main drive is not changed. Ontology terms keep their
-shared canonical paths shown above. Repeated syncs reuse the repository drive
-and retain its permissions; updating resources builds on their stored Loro
-state.
+### The imported drive, document and table
+
+Each **imported dataset** — the complete set of records one `cargo run`
+syncs, named by `API_CONSTANTS`' values (`owner=localthought,repo=test-repo-1`
+→ `localthought/test-repo-1`) — gets *one* Drive at
+`internal:/reflector-drives/<escaped-dataset>`, named from the ontology and
+dataset: for example, **github issues localthought test-repo-1**. This is one
+Drive per dataset regardless of how many nested collections it has: issue
+comments are addressed under a deeper, per-issue namespace
+(`localthought/test-repo-1/1`, one per issue) so they still dedupe correctly,
+but they file into this same Drive rather than one of their own — a document's
+complete imported dataset, comments included, never splits across drives.
+
+Under that Drive sits one **DocumentV2** (`.../document`), and under the
+document, one native AD **Table** per root-level resource the document
+declares (just `.../table/issue` for the vendored GitHub Issues document).
+A table's `classtype` is the resource's own ontology Class, so every property
+that Class recommends or requires — for `issue`, at least number, title,
+state, body, author (`user`), updated time (`updated_at`) and source URL
+(`html_url`) — renders as a column. Every `issue` record's `parent` is that
+table (so it shows up as a row); comments and other nested records keep
+`parent` pointing straight at the Drive. Every resource's `drive` property
+(used for permission/fan-out scoping, independent of `parent`) always points
+at the Drive itself.
+
+The user's main drive is not changed. Ontology terms keep their shared
+canonical paths shown above. Repeated syncs reuse the repository Drive,
+Document and Table(s) and retain their permissions; updating resources builds
+on their stored Loro state. If `DRIVE_OWNER` is set, the Drive is also added
+to that agent's saved-drive list (idempotently) on every sync, so it's
+discoverable in the app without a manual share — this only updates an
+already-locally-stored Agent resource, never fetches or invents one.
+
+**Upgrading from before this existed:** an older reflector-rs (before
+[issue #18](https://github.com/localthought/reflector-rs/issues/18)) keyed a
+Drive off each record's own namespace, so every issue's comments fragmented
+into their own separate Drive instead of sharing the dataset's one Drive. The
+next sync against an existing store migrates automatically: the one Drive
+matching the dataset itself is reused as-is, every comment gets re-pointed at
+it as it's re-synced (routine — every sync re-puts every record), and the
+leftover per-issue Drives are removed.
 
 ### Sharing AtomicServer's database on macOS
 
@@ -191,11 +227,15 @@ is not expanded by Reflector. The binary reads process environment variables;
 it does not load `.env` automatically.
 
 Set `DRIVE_OWNER` before the first sync so your AtomicServer agent can read and
-edit the new drive. Without it, no user read/write grants are added (server
+edit the new drive — the same sync also adds it to that agent's saved-drive
+list, so it shows up in the app's drive switcher without a manual share.
+Without `DRIVE_OWNER`, no user read/write grants are added (server
 administration can still access it). Reflector never makes imported private
 issues public by default. To change permissions on an existing drive, use
 AtomicServer's sharing controls. Open the drive directly at
-`<PUBLIC_URL>/reflector-drives/localthought%2Ftest-repo-1`.
+`<PUBLIC_URL>/reflector-drives/localthought%2Ftest-repo-1`, or its document at
+`<PUBLIC_URL>/reflector-drives/localthought%2Ftest-repo-1/document` for the
+issues table.
 
 ## The vendored document
 

docs/ai-logs/sessions/2026-09-07-issue-18-drive-document-table.md

@@ -0,0 +1,87 @@
+# Issue 18: user-owned drive with a document and issues table
+
+Assistant: Claude Code.
+
+## Human request
+
+Work on https://github.com/localthought/reflector-rs/issues/18, update docs
+as the work progresses, and merge at will.
+
+The issue asks for one discoverable Atomic Data drive per imported
+repository/dataset (not one per record namespace, which today fragments a
+repository's issue comments into a separate drive per issue), registered
+through the normal saved-drive mechanism; a DocumentV2 under that drive with
+a native AD table of the imported issues (number, title, state, body,
+author, updated time, source URL); comments and nested data staying
+accessible in that same drive; consistent `parent`/`drive` properties;
+reuse/idempotence across reimports and restarts; and a safe transition for
+drives created by the earlier per-namespace scheme.
+
+## Investigation
+
+Read `store.rs`'s `AtomicStorage` and `Storage::put`, `config.rs`, and
+`main.rs` to see how drives are currently created (keyed off each record's
+own `namespace`, which for a nested collection like issue comments is
+`owner/repo/<issue_number>` — one drive per issue). Confirmed against
+`syncables-rs`'s `sync::client::walk_all` that a nested collection's
+namespace is always a deeper extension of its dataset's own root namespace,
+and that root-level records are always synced (and `put`) before anything
+nested under them.
+
+Cloned `ontola/atomic-server` at the pinned revision to check the actual
+resource shapes: `urls::TABLE`'s `classtype` names a row Class (confirmed
+against `browser/data-browser/src/chunks/TablePage/createTableFromSpec.ts`
+and its AI skill doc `creating-tables.md`), a Table's rows are queried by
+`parent == table AND is-a == classtype` (`useTableData.ts`), permission
+inheritance climbs the `parent` chain while `drive` is a separate direct
+pointer used for fan-out (`lib/src/hierarchy.rs`, `resources.rs::get_drive`),
+and the saved-drive list is the Agent's `drives` property (`lib/src/db.rs`'s
+private `push_drive_to_list`, generalized here against `Storelike` since
+`AtomicStorage` isn't always a `Db`). `DocumentV2`'s real collaborative body
+is a loro-prosemirror CRDT tree with no Rust-side construction path in this
+codebase or a vendored reference implementation; `documentContent` as an
+ordinary Markdown property (the format `extract_document_plain_text`'s own
+fallback reads) was used instead of attempting to hand-build that tree.
+
+## Changes and results
+
+- `AtomicStorage` now takes a `dataset` (`with_dataset`, e.g.
+  `owner/repo`), computed by a new `Config::dataset_namespace` from
+  `API_CONSTANTS`' values in key order — one Drive per dataset, independent
+  of any individual record's own (possibly deeper) namespace.
+- Added one DocumentV2 and one native AD Table per root-level resource
+  (`ensure_document`/`ensure_table`), both idempotent and created lazily on
+  the first matching `put`. A root-level record's `parent` is now its
+  table (so it renders as a row); nested records (comments) keep `parent`
+  pointing at the drive, same drive as the root records. `drive` always
+  points at the drive itself either way.
+- `ensure_drive` now also registers the drive on `DRIVE_OWNER`'s `drives`
+  property when that agent already has a local resource, and removes any
+  drive left over from the old per-namespace scheme under this dataset
+  (`migrate_legacy_drives`) — the records that pointed at a removed drive
+  are re-pointed to the real one automatically the next time they're
+  `put`, since `persist` always rewrites `parent`/`drive` from scratch.
+- Added `html_url` to the vendored `issue` schema (`spec/`) as the issue's
+  source URL, so it flows through the existing ontology derivation as a
+  table column with no `syncables-rs` changes.
+- Updated/added `store.rs` unit tests for containment (record → table →
+  document → drive), nested collections staying off the table, saved-drive
+  registration idempotence, and legacy-drive migration; rewrote the
+  restart test's "unrelated repository" case as two dataset-scoped
+  `AtomicStorage` instances sharing one `Db`, matching how two real syncs
+  would actually share an AtomicServer database. Updated `README.md` and
+  `CLAUDE.md`.
+- `cargo fmt --check`, `cargo test` (43 tests), `cargo clippy --all-targets
+  -- -D warnings`, `cargo build`, and both `--target wasm32-unknown-unknown`
+  checks all pass locally.
+
+Not covered: this only verifies the resource shapes atomic-server's own
+Rust code and TypeScript client documented, not an actual rendered document
+open in the data browser — there is no browser or running atomic-server
+frontend in this environment to click through. `documentContent` is set as
+a Markdown summary rather than the browser's own collaborative-editor
+representation, for the reason above; the Table's `classtype` reuses the
+existing per-resource ontology Class as-is (so the table also gets
+`id`/`state_reason`/`labels` columns beyond the seven the issue names)
+rather than hand-building a second, narrower row Class duplicating those
+same property subjects. Both are called out in the PR description.

spec/github-issues.openapi.yaml

@@ -47,6 +47,10 @@ components:
         updated_at:
           type: string
           format: date-time
+        html_url:
+          type: string
+          format: uri
+          description: The issue's page on github.com — its source URL.
         user:
           $ref: '#/components/schemas/user'
         labels:

src/config.rs

@@ -261,6 +261,21 @@ impl Config {
         })
     }
 
+    /// Identifies the complete dataset this run imports — e.g.
+    /// `localthought/test-repo-1` — derived from [`Config::constants`]'
+    /// values in key order (`owner` then `repo`, for the default document).
+    /// Every record the sync writes, root or nested, is grouped under the
+    /// *one* Drive/Document/Table this names — see
+    /// [`crate::store::AtomicStorage::with_dataset`] — rather than each
+    /// record's own (possibly deeper, per-parent) namespace.
+    pub fn dataset_namespace(&self) -> String {
+        self.constants
+            .values()
+            .cloned()
+            .collect::<Vec<_>>()
+            .join("/")
+    }
+
     /// Fails early on anything the sync would only discover mid-flight: a
     /// missing document or overlay is a deployment mistake, not a sync error.
     pub fn validate(&self) -> Result<()> {
@@ -355,6 +370,16 @@ mod tests {
         assert_eq!(constants["repo"], "test-repo-1");
     }
 
+    #[test]
+    fn dataset_namespace_joins_constant_values_in_key_order() {
+        let config = Config::from_lookup(Path::new("/reflector"), |key| match key {
+            env_var::PUBLIC_URL => Some("http://localhost:9883".into()),
+            _ => None,
+        })
+        .unwrap();
+        assert_eq!(config.dataset_namespace(), "localthought/test-repo-1");
+    }
+
     #[test]
     fn public_url_loses_its_trailing_slash() {
         assert_eq!(

src/main.rs

@@ -68,7 +68,8 @@ async fn main() -> Result<()> {
         Arc::new(store),
         SubjectMapper::new(config.public_url.clone()),
     )
-    .with_drive_owner(config.drive_owner.clone());
+    .with_drive_owner(config.drive_owner.clone())
+    .with_dataset(config.dataset_namespace());
 
     let client = SyncClient::new(
         ClientConfig {