diff --git a/docs/level-migration.md b/docs/level-migration.md new file mode 100644 index 00000000..e9480aed --- /dev/null +++ b/docs/level-migration.md @@ -0,0 +1,141 @@ +# Implementation plan: migrate `leveldown` → `abstract-level` family + +## Goal + +Replace the legacy Level stack (`leveldown`, `levelup`, `memdown`, +`subleveldown`, `encoding-down`, `abstract-leveldown`, `deferred-leveldown`) +with the actively maintained `abstract-level` family. The on-disk LevelDB +format is unchanged, so **no data migration is required** — `classic-level` +opens existing `leveldown` databases. + +Sublevel on-disk compatibility confirmed: `abstract-level/UPGRADING.md` +states the sublevel key structure is equal to that of `subleveldown`, so an +`abstract-level` sublevel reads data previously written by `subleveldown`. + +## Package changes + +- **Remove:** `leveldown`, `levelup`, `memdown`, `subleveldown`, + `encoding-down`, `abstract-leveldown`, `deferred-leveldown` +- **Add:** `classic-level` (native, N-API), `memory-level`, `abstract-level` + +## Guiding principle + +Keep the exported `L.*` helper API in `src/shared/level/index.js` +**unchanged**. Then the ~21 consumer files stay untouched and the migration +is confined to 6 files. + +## API differences to handle + +| Old | New | +|---|---| +| `levelup(encode(leveldown(loc), enc))` | `new ClassicLevel(loc, { valueEncoding })` | +| `subleveldown(db, name, enc)` | `db.sublevel(name, { valueEncoding })` | +| `db.get(k)` throws `NotFound` | `db.get(k)` returns `undefined` | +| `db.createReadStream(opts)` | `db.iterator(opts)` / `db.keys()` / `db.values()` (async-iterable) | +| `AbstractLevelDOWN` / `AbstractIterator` (callback) | `AbstractLevel` / `AbstractIterator` (promise-based) | +| `getMany`, `batch`, range options (`gte/lte/limit/reverse`) | unchanged | + +## Phases + +### Phase 0 — Branch & preparation +- Branch `chore/level-migration`. +- Install the new packages *alongside* the old ones, so the migration can + proceed file by file with tests staying green. + +### Phase 1 — WKB regression tests (done) +- The `wkb.js` encoding-format conversion cannot be done in isolation: the + exported encoding object is consumed by `index.js`, and the old + (`encoding-down`: `{ buffer, encode, decode }`) and new + (`abstract-level` / `level-transcoder`: `{ name, format, encode, decode }`) + shapes are incompatible. The `wkb.js` code change therefore moves into + Phase 2, alongside its consumer. +- Phase 1 delivers the safety net: expand `test/shared/level/wkb-test.js` + on the old stack — every geometry type plus batch/getMany/iterator/del — + so Phase 2 can be verified against identical behaviour. + +### Phase 2 — `index.js` + `wkb.js` (factory, helpers, encoding) +- New `leveldb()` factory: `classic-level` / `memory-level` instead of the + `levelup`/`encode`/`leveldown` nesting; `sublevel` branches → `db.sublevel()`. +- Port the WKB encoding in `wkb.js` to the `abstract-level` custom-encoding + shape (`{ name, format, encode, decode }`, via `level-transcoder`). Open + question: current `format` (`'buffer'` vs `'view'`). +- Rewrite the stream readers (`read`, `Streams`, `readStream`, + `readTuples/Keys/Values`, `existsKey`) on async iterators. +- `get()` helper: check for `undefined` instead of `try/catch` on `NotFound`. +- **Keep the exported signatures identical.** Verify with the Phase 1 tests. + +**Findings from the phase 2 investigation:** +- The `leveldb({ down })` factory path is used by `Store.js` + (`PartitionDOWN`) and `ipc-test.js` (`IPCDownClient`), so it couples + phase 2 to phases 3/4. Mitigation: keep `levelup` (still installed from + phase 0) as a temporary bridge for the `down:` path only, until phases + 3/4 replace the custom stores. +- `abstract-level` changed the event model: there is **no** `put` / `del` + / `batch` event anymore, only a single `write` event carrying an + `operations` array. This affects `PreferencesStore.js` (`on('put'/'del')`), + `SearchIndex.js` (`on('del'/'batch')`) and `SpatialIndex.js` + (`on('batch')`). These listeners attach directly to the db objects, so + they are not covered by the "consumers unchanged via `L.*`" principle and + must be migrated explicitly (phase 2b). + +### Phase 2b — event-model migration +- Adapt `PreferencesStore.js`, `SearchIndex.js`, `SpatialIndex.js` from the + `put`/`del`/`batch` events to the single `write` event (filter the + `operations` array by `type`). + +**Coupling finding (phases 2/3/4 cannot land separately with green tests):** +Once `index.js` is migrated, the child databases passed to `PartitionDOWN` +and `IPCServer` are `abstract-level` instances. Their iterator API is +promise-based (`for await`, `iterator.next()` returns `[key, value]`), +incompatible with the callback `next(cb)` style the custom stores still +use. The `levelup` bridge only wraps the *outer* `down:` database; it does +not help the custom store talk to its *children*. Phases 3 and 4 must +therefore land together with phase 2 to reach a green test suite. + +### Phase 3 — `PartitionDOWN.js` (core piece) +- Reimplement as an `AbstractLevel` subclass delegating to two child DBs + (JSON + WKB). +- Private methods on the promise contract: + `_open/_close/_get/_getMany/_put/_del/_batch/_iterator`. +- Custom iterator: port the two-iterator synchronisation logic to the new + `_next() → [key, value]` model. +- Adapt `test/shared/level/PartitionDOWN-test.js`, keep green. + +### Phase 4 — `ipc.js` (renderer↔main bridge) +- `IPCDownClient` → `AbstractLevel` subclass; `IPCIterator` → new + `AbstractIterator`. +- `IPCServer`: replace `db.createReadStream` in the `ITERATOR` handler with + `db.iterator()`. +- Option considered but **not recommended**: `many-level` instead of the + hand-rolled bridge — would need an IPC↔duplex-stream adapter, more moving + parts. +- Adapt `test/shared/level/ipc-test.js`, keep green. + +### Phase 5 — direct imports +- `src/renderer/components/ProjectList-services.js` + (`levelup`/`memdown`/`subleveldown`). +- `src/main/legacy/transfer.js`, `src/main/preload/preload.js`. + +### Phase 6 — cleanup +- Remove the old packages from `package.json`, `npm install`, verify the + lockfile. + +### Phase 7 — verification +- `npm run lint`, `npm test` (especially `test/shared/level/*`, + `test/renderer/store/schema/`, `test/main/stores/`). +- Manual (clean build): open/create/save a project, features with geometry + (the PartitionDOWN path), schema upgrade, replication, legacy transfer. + +## Risks + +- **Storage core** — data-path bugs are severe; the tests are the safety + net (present for all three `shared/level` files). +- **PartitionDOWN iterator** — the two-iterator synchronisation is the + trickiest part. +- **IPC iterator** — currently fetches the whole result at once; keep that + behaviour deliberately, or deliberately switch to real streaming (record + the decision). + +## Effort + +~1–1.5 days of focused work including tests. diff --git a/package-lock.json b/package-lock.json index 11ba54b1..c3b4de49 100644 --- a/package-lock.json +++ b/package-lock.json @@ -12,11 +12,12 @@ "dependencies": { "@mdi/js": "^7.0.96", "@mdi/react": "^1.6.0", - "@syncpoint/matrix-client-api": "^2.3.0", + "@syncpoint/matrix-client-api": "^3.0.0", "@syncpoint/signal": "^1.3.0", "@syncpoint/signs": "^1.1.0", "@syncpoint/wkx": "^0.5.2", - "abstract-leveldown": "^7.2.0", + "abstract-level": "^3.1.1", + "classic-level": "^3.0.0", "color": "^5.0.2", "dotenv": "^17.2.3", "fuse.js": "^7.1.0", @@ -26,9 +27,8 @@ "jspdf": "^4.2.1", "jsts": "^2.12.1", "kbar": "^0.1.0-beta.43", - "leveldown": "^6.1.1", - "levelup": "^5.0.1", "luxon": "^3.7.2", + "memory-level": "^3.1.0", "minisearch": "^7.2.0", "mousetrap": "^1.6.5", "mousetrap-global-bind": "^1.1.0", @@ -45,7 +45,6 @@ "react-tooltip": "^5.29.1", "reproject": "^1.2.7", "sanitize-filename": "^1.6.3", - "subleveldown": "^6.0.1", "throttle-debounce": "^5.0.2", "typeface-roboto": "^1.1.13", "uniqolor": "^1.1.1" @@ -68,7 +67,6 @@ "eslint-plugin-react-hooks": "^5.2.0", "file-loader": "^6.2.0", "html-webpack-plugin": "^5.6.4", - "memdown": "^6.1.1", "mocha": "^11.7.5", "source-map-loader": "^5.0.0", "style-loader": "^4.0.0", @@ -3532,9 +3530,9 @@ } }, "node_modules/@syncpoint/matrix-client-api": { - "version": "2.3.0", - "resolved": "https://registry.npmjs.org/@syncpoint/matrix-client-api/-/matrix-client-api-2.3.0.tgz", - "integrity": "sha512-s/i+Xz76wuu514E9v/biUhEAxMAqTxjCHi4GsGv8TS8dsw4UfpqB80A5eLtxpurE01U/3r/7z2181ZbMc3Ut7w==", + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/@syncpoint/matrix-client-api/-/matrix-client-api-3.0.0.tgz", + "integrity": "sha512-8GNHKYfSb+m88YrwYBcNfoowx94bMFTxrFg9HHSJkHYEBy9aAA2lHlTiGR1LjHqb6l/pkr/hWVxcoA6KMZMNjg==", "license": "MIT", "dependencies": { "@matrix-org/matrix-sdk-crypto-wasm": "^17.1.0", @@ -4221,22 +4219,30 @@ "node": "^18.17.0 || >=20.5.0" } }, - "node_modules/abstract-leveldown": { - "version": "7.2.0", - "resolved": "https://registry.npmjs.org/abstract-leveldown/-/abstract-leveldown-7.2.0.tgz", - "integrity": "sha512-DnhQwcFEaYsvYDnACLZhMmCWd3rkOeEvglpa4q5i/5Jlm3UIsWaxVzuXvDLFCSCWRO3yy2/+V/G7FusFgejnfQ==", - "deprecated": "Superseded by abstract-level (https://github.com/Level/community#faq)", + "node_modules/abstract-level": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/abstract-level/-/abstract-level-3.1.1.tgz", + "integrity": "sha512-CW2gKbJFTuX1feMvOrvsVMmijAOgI9kg2Ie9Dq3gOcMt/dVVoVmqNlLcEUCT13NxHFMEajcUcVBIplbyDroDiw==", "license": "MIT", "dependencies": { "buffer": "^6.0.3", - "catering": "^2.0.0", "is-buffer": "^2.0.5", - "level-concat-iterator": "^3.0.0", - "level-supports": "^2.0.1", - "queue-microtask": "^1.2.3" + "level-supports": "^6.2.0", + "level-transcoder": "^1.0.1", + "maybe-combine-errors": "^1.0.0", + "module-error": "^1.0.1" }, "engines": { - "node": ">=10" + "node": ">=18" + } + }, + "node_modules/abstract-level/node_modules/level-supports": { + "version": "6.2.0", + "resolved": "https://registry.npmjs.org/level-supports/-/level-supports-6.2.0.tgz", + "integrity": "sha512-QNxVXP0IRnBmMsJIh+sb2kwNCYcKciQZJEt+L1hPCHrKNELllXhvrlClVHXBYZVT+a7aTSM6StgNXdAldoab3w==", + "license": "MIT", + "engines": { + "node": ">=16" } }, "node_modules/accepts": { @@ -5759,15 +5765,6 @@ "node": ">=10.0.0" } }, - "node_modules/catering": { - "version": "2.1.1", - "resolved": "https://registry.npmjs.org/catering/-/catering-2.1.1.tgz", - "integrity": "sha512-K7Qy8O9p76sL3/3m7/zLKbRkyOlSZAgzEaLhyj2mXS8PsCud2Eo4hAb8aLtZqHh0QGqLcb9dlJSu6lHRVENm1w==", - "license": "MIT", - "engines": { - "node": ">=6" - } - }, "node_modules/chalk": { "version": "4.1.2", "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz", @@ -5844,6 +5841,28 @@ "node": ">=8" } }, + "node_modules/classic-level": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/classic-level/-/classic-level-3.0.0.tgz", + "integrity": "sha512-yGy8j8LjPbN0Bh3+ygmyYvrmskVita92pD/zCoalfcC9XxZj6iDtZTAnz+ot7GG8p9KLTG+MZ84tSA4AhkgVZQ==", + "hasInstallScript": true, + "license": "MIT", + "dependencies": { + "abstract-level": "^3.1.0", + "module-error": "^1.0.1", + "napi-macros": "^2.2.2", + "node-gyp-build": "^4.3.0" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/classic-level/node_modules/napi-macros": { + "version": "2.2.2", + "resolved": "https://registry.npmjs.org/napi-macros/-/napi-macros-2.2.2.tgz", + "integrity": "sha512-hmEVtAGYzVQpCKdbQea4skABsdXW4RUh5t5mJ2zzqowJS2OyXZTU1KhDVFhx+NlWZ4ap9mqR9TcDO3LTTttd+g==", + "license": "MIT" + }, "node_modules/classnames": { "version": "2.5.1", "resolved": "https://registry.npmjs.org/classnames/-/classnames-2.5.1.tgz", @@ -6552,20 +6571,6 @@ "node": ">=10" } }, - "node_modules/deferred-leveldown": { - "version": "7.0.0", - "resolved": "https://registry.npmjs.org/deferred-leveldown/-/deferred-leveldown-7.0.0.tgz", - "integrity": "sha512-QKN8NtuS3BC6m0B8vAnBls44tX1WXAFATUsJlruyAYbZpysWV3siH6o/i3g9DCHauzodksO60bdj5NazNbjCmg==", - "deprecated": "Superseded by abstract-level (https://github.com/Level/community#faq)", - "license": "MIT", - "dependencies": { - "abstract-leveldown": "^7.2.0", - "inherits": "^2.0.3" - }, - "engines": { - "node": ">=10" - } - }, "node_modules/define-data-property": { "version": "1.1.4", "resolved": "https://registry.npmjs.org/define-data-property/-/define-data-property-1.1.4.tgz", @@ -6615,12 +6620,6 @@ "url": "https://github.com/sponsors/ljharb" } }, - "node_modules/defined": { - "version": "0.0.0", - "resolved": "https://registry.npmjs.org/defined/-/defined-0.0.0.tgz", - "integrity": "sha512-zpqiCT8bODLu3QSmLLic8xJnYWBFjOSu/fBCm189oAiTtPq/PSanNACKZDS7kgSyCJY7P+IcODzlIogBK/9RBg==", - "license": "MIT" - }, "node_modules/delayed-stream": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz", @@ -7323,22 +7322,6 @@ "iconv-lite": "^0.6.2" } }, - "node_modules/encoding-down": { - "version": "7.1.0", - "resolved": "https://registry.npmjs.org/encoding-down/-/encoding-down-7.1.0.tgz", - "integrity": "sha512-ky47X5jP84ryk5EQmvedQzELwVJPjCgXDQZGeb9F6r4PdChByCGHTBrVcF3h8ynKVJ1wVbkxTsDC8zBROPypgQ==", - "deprecated": "Superseded by abstract-level (https://github.com/Level/community#faq)", - "license": "MIT", - "dependencies": { - "abstract-leveldown": "^7.2.0", - "inherits": "^2.0.3", - "level-codec": "^10.0.0", - "level-errors": "^3.0.0" - }, - "engines": { - "node": ">=10" - } - }, "node_modules/end-of-stream": { "version": "1.4.5", "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz", @@ -8871,7 +8854,6 @@ "version": "1.0.1", "resolved": "https://registry.npmjs.org/functional-red-black-tree/-/functional-red-black-tree-1.0.1.tgz", "integrity": "sha512-dsKNQNdj6xA3T+QlADDA7mOSlX0qiMINjn0cgr+eGHGsbSHzTabcIogz2+p/iqP1Xs6EP/sS2SbqH+brGTbq0g==", - "dev": true, "license": "MIT" }, "node_modules/functions-have-names": { @@ -11019,105 +11001,17 @@ "integrity": "sha512-Rm4J/WaHhRa93nCN2mwWDZFoRVF18G1f47C+kvQWyHGEZxFpTUi73p7lMVSAndyxGt6lJ2/CFbOcf9ra5p8aww==", "license": "Apache-2.0" }, - "node_modules/level-codec": { - "version": "10.0.0", - "resolved": "https://registry.npmjs.org/level-codec/-/level-codec-10.0.0.tgz", - "integrity": "sha512-QW3VteVNAp6c/LuV6nDjg7XDXx9XHK4abmQarxZmlRSDyXYk20UdaJTSX6yzVvQ4i0JyWSB7jert0DsyD/kk6g==", - "deprecated": "Superseded by level-transcoder (https://github.com/Level/community#faq)", - "license": "MIT", - "dependencies": { - "buffer": "^6.0.3" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/level-concat-iterator": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/level-concat-iterator/-/level-concat-iterator-3.1.0.tgz", - "integrity": "sha512-BWRCMHBxbIqPxJ8vHOvKUsaO0v1sLYZtjN3K2iZJsRBYtp+ONsY6Jfi6hy9K3+zolgQRryhIn2NRZjZnWJ9NmQ==", - "deprecated": "Superseded by abstract-level (https://github.com/Level/community#faq)", - "license": "MIT", - "dependencies": { - "catering": "^2.1.0" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/level-errors": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/level-errors/-/level-errors-3.0.1.tgz", - "integrity": "sha512-tqTL2DxzPDzpwl0iV5+rBCv65HWbHp6eutluHNcVIftKZlQN//b6GEnZDM2CvGZvzGYMwyPtYppYnydBQd2SMQ==", - "deprecated": "Superseded by abstract-level (https://github.com/Level/community#faq)", - "license": "MIT", - "engines": { - "node": ">=10" - } - }, - "node_modules/level-iterator-stream": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/level-iterator-stream/-/level-iterator-stream-5.0.0.tgz", - "integrity": "sha512-wnb1+o+CVFUDdiSMR/ZymE2prPs3cjVLlXuDeSq9Zb8o032XrabGEXcTCsBxprAtseO3qvFeGzh6406z9sOTRA==", - "license": "MIT", - "dependencies": { - "inherits": "^2.0.4", - "readable-stream": "^3.4.0" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/level-option-wrap": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/level-option-wrap/-/level-option-wrap-1.1.0.tgz", - "integrity": "sha512-gQouC22iCqHuBLNl4BHxEZUxLvUKALAtT/Q0c6ziOxZQ8c02G/gyxHWNbLbxUzRNfMrRnbt6TZT3gNe8VBqQeg==", - "license": "MIT", - "dependencies": { - "defined": "~0.0.0" - } - }, - "node_modules/level-supports": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/level-supports/-/level-supports-2.1.0.tgz", - "integrity": "sha512-E486g1NCjW5cF78KGPrMDRBYzPuueMZ6VBXHT6gC7A8UYWGiM14fGgp+s/L1oFfDWSPV/+SFkYCmZ0SiESkRKA==", - "license": "MIT", - "engines": { - "node": ">=10" - } - }, - "node_modules/leveldown": { - "version": "6.1.1", - "resolved": "https://registry.npmjs.org/leveldown/-/leveldown-6.1.1.tgz", - "integrity": "sha512-88c+E+Eizn4CkQOBHwqlCJaTNEjGpaEIikn1S+cINc5E9HEvJ77bqY4JY/HxT5u0caWqsc3P3DcFIKBI1vHt+A==", - "deprecated": "Superseded by classic-level (https://github.com/Level/community#faq)", - "hasInstallScript": true, - "license": "MIT", - "dependencies": { - "abstract-leveldown": "^7.2.0", - "napi-macros": "~2.0.0", - "node-gyp-build": "^4.3.0" - }, - "engines": { - "node": ">=10.12.0" - } - }, - "node_modules/levelup": { - "version": "5.1.1", - "resolved": "https://registry.npmjs.org/levelup/-/levelup-5.1.1.tgz", - "integrity": "sha512-0mFCcHcEebOwsQuk00WJwjLI6oCjbBuEYdh/RaRqhjnyVlzqf41T1NnDtCedumZ56qyIh8euLFDqV1KfzTAVhg==", - "deprecated": "Superseded by abstract-level (https://github.com/Level/community#faq)", + "node_modules/level-transcoder": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/level-transcoder/-/level-transcoder-1.0.1.tgz", + "integrity": "sha512-t7bFwFtsQeD8cl8NIoQ2iwxA0CL/9IFw7/9gAjOonH0PWTTiRfY7Hq+Ejbsxh86tXobDQ6IOiddjNYIfOBs06w==", "license": "MIT", "dependencies": { - "catering": "^2.0.0", - "deferred-leveldown": "^7.0.0", - "level-errors": "^3.0.1", - "level-iterator-stream": "^5.0.0", - "level-supports": "^2.0.1", - "queue-microtask": "^1.2.3" + "buffer": "^6.0.3", + "module-error": "^1.0.1" }, "engines": { - "node": ">=10" + "node": ">=12" } }, "node_modules/levn": { @@ -11276,13 +11170,6 @@ "yallist": "^3.0.2" } }, - "node_modules/ltgt": { - "version": "2.2.1", - "resolved": "https://registry.npmjs.org/ltgt/-/ltgt-2.2.1.tgz", - "integrity": "sha512-AI2r85+4MquTw9ZYqabu4nMwy9Oftlfa/e/52t9IjtfG+mGBbTNdAoZ3RQKLHR6r0wQnwZnPIEh/Ya6XTWAKNA==", - "dev": true, - "license": "MIT" - }, "node_modules/luxon": { "version": "3.7.2", "resolved": "https://registry.npmjs.org/luxon/-/luxon-3.7.2.tgz", @@ -11379,6 +11266,15 @@ "node": ">= 0.4" } }, + "node_modules/maybe-combine-errors": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/maybe-combine-errors/-/maybe-combine-errors-1.0.0.tgz", + "integrity": "sha512-eefp6IduNPT6fVdwPp+1NgD0PML1NU5P6j1Mj5nz1nidX8/sWY7119WL8vTAHgqfsY74TzW0w1XPgdYEKkGZ5A==", + "license": "MIT", + "engines": { + "node": ">=10" + } + }, "node_modules/media-typer": { "version": "0.3.0", "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz", @@ -11389,24 +11285,6 @@ "node": ">= 0.6" } }, - "node_modules/memdown": { - "version": "6.1.1", - "resolved": "https://registry.npmjs.org/memdown/-/memdown-6.1.1.tgz", - "integrity": "sha512-vh2RiuVrn6Vv73088C1KzLwy9+hhRwoZsgddYqIoVuFFrcoc2Rt+lq/KrmkFn6ulko7AtQ0AvqtYid35exb38A==", - "deprecated": "Superseded by memory-level (https://github.com/Level/community#faq)", - "dev": true, - "license": "MIT", - "dependencies": { - "abstract-leveldown": "^7.2.0", - "buffer": "^6.0.3", - "functional-red-black-tree": "^1.0.1", - "inherits": "^2.0.1", - "ltgt": "^2.2.0" - }, - "engines": { - "node": ">=10" - } - }, "node_modules/memfs": { "version": "4.56.10", "resolved": "https://registry.npmjs.org/memfs/-/memfs-4.56.10.tgz", @@ -11437,6 +11315,20 @@ "tslib": "2" } }, + "node_modules/memory-level": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/memory-level/-/memory-level-3.1.0.tgz", + "integrity": "sha512-mTqFVi5iReKcjue/pag0OY4VNU7dlagCyjjPwWGierpk1Bpl9WjOxgXIswymPW3Q9bj3Foay+Z16mPGnKzvTkQ==", + "license": "MIT", + "dependencies": { + "abstract-level": "^3.1.0", + "functional-red-black-tree": "^1.0.1", + "module-error": "^1.0.1" + }, + "engines": { + "node": ">=18" + } + }, "node_modules/merge-descriptors": { "version": "1.0.3", "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.3.tgz", @@ -11859,6 +11751,15 @@ "url": "https://github.com/chalk/supports-color?sponsor=1" } }, + "node_modules/module-error": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/module-error/-/module-error-1.0.2.tgz", + "integrity": "sha512-0yuvsqSCv8LbaOKhnsQ/T5JhyFlCYLPXK3U2sgV10zoKQwzs/MyfuQUOZQ1V/6OCOJsK/TRgNVrPuPDqtdMFtA==", + "license": "MIT", + "engines": { + "node": ">=10" + } + }, "node_modules/mousetrap": { "version": "1.6.5", "resolved": "https://registry.npmjs.org/mousetrap/-/mousetrap-1.6.5.tgz", @@ -11914,12 +11815,6 @@ "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" } }, - "node_modules/napi-macros": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/napi-macros/-/napi-macros-2.0.0.tgz", - "integrity": "sha512-A0xLykHtARfueITVDernsAWdtIMbOJgKgcluwENp3AlsKN/PloyO10HtmoqnFAQAcxPkgZN7wdfPfEd0zNGxbg==", - "license": "MIT" - }, "node_modules/natural-compare": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/natural-compare/-/natural-compare-1.4.0.tgz", @@ -13217,6 +13112,7 @@ "version": "1.2.3", "resolved": "https://registry.npmjs.org/queue-microtask/-/queue-microtask-1.2.3.tgz", "integrity": "sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A==", + "dev": true, "funding": [ { "type": "github", @@ -13231,7 +13127,8 @@ "url": "https://feross.org/support" } ], - "license": "MIT" + "license": "MIT", + "peer": true }, "node_modules/quick-lru": { "version": "5.1.1", @@ -13320,15 +13217,6 @@ "quickselect": "^3.0.0" } }, - "node_modules/reachdown": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/reachdown/-/reachdown-1.1.0.tgz", - "integrity": "sha512-6LsdRe4cZyOjw4NnvbhUd/rGG7WQ9HMopPr+kyL018Uci4kijtxcGR5kVb5Ln13k4PEE+fEFQbjfOvNw7cnXmA==", - "license": "MIT", - "engines": { - "node": ">=6" - } - }, "node_modules/react": { "version": "18.3.1", "resolved": "https://registry.npmjs.org/react/-/react-18.3.1.tgz", @@ -14915,24 +14803,6 @@ "webpack": "^5.27.0" } }, - "node_modules/subleveldown": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/subleveldown/-/subleveldown-6.0.1.tgz", - "integrity": "sha512-Cnf+cn2wISXU2xflY1SFIqfX4hG2d6lFk2P5F8RDQLmiqN9Ir4ExNfUFH6xnmizMseM/t+nMsDUKjN9Kw6ShFA==", - "deprecated": "Superseded by abstract-level (https://github.com/Level/community#faq)", - "license": "MIT", - "dependencies": { - "abstract-leveldown": "^7.2.0", - "encoding-down": "^7.1.0", - "inherits": "^2.0.3", - "level-option-wrap": "^1.1.0", - "levelup": "^5.1.1", - "reachdown": "^1.1.0" - }, - "engines": { - "node": ">=10" - } - }, "node_modules/sumchecker": { "version": "3.0.1", "resolved": "https://registry.npmjs.org/sumchecker/-/sumchecker-3.0.1.tgz", diff --git a/package.json b/package.json index 42d27f4f..7ddb2d47 100644 --- a/package.json +++ b/package.json @@ -48,7 +48,6 @@ "eslint-plugin-react-hooks": "^5.2.0", "file-loader": "^6.2.0", "html-webpack-plugin": "^5.6.4", - "memdown": "^6.1.1", "mocha": "^11.7.5", "source-map-loader": "^5.0.0", "style-loader": "^4.0.0", @@ -68,11 +67,12 @@ "dependencies": { "@mdi/js": "^7.0.96", "@mdi/react": "^1.6.0", - "@syncpoint/matrix-client-api": "^2.3.0", + "@syncpoint/matrix-client-api": "^3.0.0", "@syncpoint/signal": "^1.3.0", "@syncpoint/signs": "^1.1.0", "@syncpoint/wkx": "^0.5.2", - "abstract-leveldown": "^7.2.0", + "abstract-level": "^3.1.1", + "classic-level": "^3.0.0", "color": "^5.0.2", "dotenv": "^17.2.3", "fuse.js": "^7.1.0", @@ -82,9 +82,8 @@ "jspdf": "^4.2.1", "jsts": "^2.12.1", "kbar": "^0.1.0-beta.43", - "leveldown": "^6.1.1", - "levelup": "^5.0.1", "luxon": "^3.7.2", + "memory-level": "^3.1.0", "minisearch": "^7.2.0", "mousetrap": "^1.6.5", "mousetrap-global-bind": "^1.1.0", @@ -101,7 +100,6 @@ "react-tooltip": "^5.29.1", "reproject": "^1.2.7", "sanitize-filename": "^1.6.3", - "subleveldown": "^6.0.1", "throttle-debounce": "^5.0.2", "typeface-roboto": "^1.1.13", "uniqolor": "^1.1.1" diff --git a/src/main/legacy/transfer.js b/src/main/legacy/transfer.js index 035443ca..d2c480d0 100644 --- a/src/main/legacy/transfer.js +++ b/src/main/legacy/transfer.js @@ -9,7 +9,7 @@ import * as L from '../../shared/level' * Project databases are only created once from main process and * are then used only by renderer. * - * @param {*} db plain leveldown instance, i.e. without encodings + * @param {*} db plain abstract-level instance, i.e. without encodings * @param {*} project project to transfer */ export const transferProject = async (db, project) => { diff --git a/src/main/preload/preload.js b/src/main/preload/preload.js index fe799413..52a517f9 100644 --- a/src/main/preload/preload.js +++ b/src/main/preload/preload.js @@ -8,7 +8,7 @@ const preferences = require('./modules/preferences') const editing = require('./modules/editing') const platform = require('./modules/platform') -// contextIsolation is off (renderer needs nodeIntegration for leveldown), +// contextIsolation is off (renderer needs nodeIntegration for classic-level), // so we assign directly instead of using contextBridge. window.odin = { projects, diff --git a/src/renderer/components/Project-services.js b/src/renderer/components/Project-services.js index 48e24021..5a09aac1 100644 --- a/src/renderer/components/Project-services.js +++ b/src/renderer/components/Project-services.js @@ -160,7 +160,7 @@ export default async projectUUID => { services.replicationProvider = MatrixClient({ ...credentials, device_id: projectUUID, - db: L.leveldb({ up: db, encoding: 'json', prefix: 'command-queue' }), + db: L.leveldb({ parent: db, encoding: 'json', prefix: 'command-queue' }), ...(encryption && { encryption }) }) } else { diff --git a/src/renderer/components/ProjectList-services.js b/src/renderer/components/ProjectList-services.js index deccd0b6..87eb341d 100644 --- a/src/renderer/components/ProjectList-services.js +++ b/src/renderer/components/ProjectList-services.js @@ -1,6 +1,4 @@ -import levelup from 'levelup' -import memdown from 'memdown' -import sublevel from 'subleveldown' +import * as L from '../../shared/level' import ProjectStore from '../store/ProjectStore' import { Selection } from '../Selection' import { MatrixClient } from '@syncpoint/matrix-client-api' @@ -17,7 +15,7 @@ export default async () => { ? MatrixClient({ ...credentials, device_id: 'PROJECT-LIST', - db: sublevel(levelup(memdown()), 'command-queue', { valueEncoding: 'json' }) + db: L.leveldb({ parent: L.leveldb(), encoding: 'json', prefix: 'command-queue' }) }) : { disabled: true diff --git a/src/renderer/store/PreferencesStore.js b/src/renderer/store/PreferencesStore.js index 61c4fb00..28456ad4 100644 --- a/src/renderer/store/PreferencesStore.js +++ b/src/renderer/store/PreferencesStore.js @@ -15,14 +15,18 @@ export default function PreferencesStore (preferencesDB, prefsBridge) { prefsBridge.putAll(tuples) })() - preferencesDB.on('put', (key, value) => { - prefsBridge.post(key, value) - this.emit(key, { value }) - }) - - preferencesDB.on('del', key => { - prefsBridge.del(key) - this.emit(key, { value: undefined }) + // abstract-level emits a single 'write' event carrying an operations + // array for put/del/batch alike. + preferencesDB.on('write', operations => { + operations.forEach(op => { + if (op.type === 'put') { + prefsBridge.post(op.key, op.value) + this.emit(op.key, { value: op.value }) + } else { + prefsBridge.del(op.key) + this.emit(op.key, { value: undefined }) + } + }) }) this.unsubscribers = [ diff --git a/src/renderer/store/SearchIndex.js b/src/renderer/store/SearchIndex.js index 45367d12..10bab6c8 100644 --- a/src/renderer/store/SearchIndex.js +++ b/src/renderer/store/SearchIndex.js @@ -74,9 +74,9 @@ SearchIndex.prototype.bootstrap = async function () { documents.forEach(doc => (this.cachedDocuments[doc.id] = doc)) this.index.addAll(documents) - // Register store listeners: - this.jsonDB.on('del', key => this.handleBatch([{ type: 'del', key }])) - this.jsonDB.on('batch', event => this.handleBatch(event)) + // Register store listener. abstract-level emits a single 'write' event + // carrying an operations array for put/del/batch alike. + this.jsonDB.on('write', operations => this.handleBatch(operations)) } diff --git a/src/renderer/store/SessionStore.js b/src/renderer/store/SessionStore.js index 7ca32f79..5d717c1f 100644 --- a/src/renderer/store/SessionStore.js +++ b/src/renderer/store/SessionStore.js @@ -7,8 +7,13 @@ export default function SessionStore (db) { this.db = db this.cache = {} - this.db.on('put', (key, value) => this.emit('put', { key, value })) - this.db.on('del', (key) => this.emit('del', { key })) + // abstract-level emits a single 'write' event carrying an operations + // array for put/del/batch alike. + this.db.on('write', operations => { + operations.forEach(op => op.type === 'put' + ? this.emit('put', { key: op.key, value: op.value }) + : this.emit('del', { key: op.key })) + }) } util.inherits(SessionStore, Emitter) diff --git a/src/renderer/store/SpatialIndex.js b/src/renderer/store/SpatialIndex.js index 37c90fbe..b84a6aea 100644 --- a/src/renderer/store/SpatialIndex.js +++ b/src/renderer/store/SpatialIndex.js @@ -12,7 +12,7 @@ export function SpatialIndex (wkbDB) { this.tree = new RBush() this.geoJSONReader = new TS.GeoJSONReader() - wkbDB.on('batch', this.update.bind(this)) + wkbDB.on('write', this.update.bind(this)) } diff --git a/src/renderer/store/Store.js b/src/renderer/store/Store.js index a1592127..38bb1458 100644 --- a/src/renderer/store/Store.js +++ b/src/renderer/store/Store.js @@ -3,7 +3,7 @@ import * as R from 'ramda' import Emitter from '../../shared/emitter' import * as ID from '../ids' import * as L from '../../shared/level' -import { PartitionDOWN } from '../../shared/level/PartitionDOWN' +import { PartitionStore } from '../../shared/level/PartitionStore' import * as TS from '../ol/ts' import { transform, geometryType } from '../model/geometry' import { readGeometry } from '../ol/format' @@ -57,7 +57,7 @@ export default function Store (jsonDB, wkbDB, undo, selection) { this.wkbDB = wkbDB this.undo = undo this.selection = selection - this.db = L.leveldb({ down: new PartitionDOWN(jsonDB, wkbDB) }) + this.db = new PartitionStore(jsonDB, wkbDB) } util.inherits(Store, Emitter) diff --git a/src/shared/level/PartitionDOWN.js b/src/shared/level/PartitionDOWN.js deleted file mode 100644 index e42129ee..00000000 --- a/src/shared/level/PartitionDOWN.js +++ /dev/null @@ -1,252 +0,0 @@ -import util from 'util' -import { AbstractLevelDOWN, AbstractIterator } from 'abstract-leveldown' - -/** - * - */ -function Iterator (db, options) { - AbstractIterator.call(this, db) - this.options_ = options // remember if keys are requested - this.properties = options.properties - this.geometry = options.geometry - - // synched :: Boolean - // Whether properties/geometry keys are in sync. - this.insync = true -} - -util.inherits(Iterator, AbstractIterator) - -const next = it => new Promise((resolve, reject) => { - it.next((err, key, value) => { - if (err) reject(err) - else resolve({ key, value }) - }) -}) - -const consumed = it => it.key === undefined - -Iterator.prototype._next = async function (callback) { - - try { - // Fetch properties unconditionally and geometry only when keys matched. - this.propertiesKV = await next(this.properties) - if (this.insync) this.geometryKV = await next(this.geometry) - - // We are done, when both iterators are consumed. - if (consumed(this.geometryKV) && consumed(this.propertiesKV)) { - return this._nextTick(callback) - } - - this.insync = this.propertiesKV.key === this.geometryKV.key - - const key = this.propertiesKV.key - const value = this.insync - ? { ...this.propertiesKV.value, geometry: this.geometryKV.value } - : this.propertiesKV.value - - this._nextTick(callback, null, key, value) - } catch (err) { - this._nextTick(callback, err) - } -} - - -/** - * AbstractLevelDOWN which splits values into two different databases. - * The value's optional `geometry` property is encoded as WKB to `wkbDB`. - * All other properties are written as JSON to `jsonDB`. - */ -export const PartitionDOWN = function (jsonDB, wkbDB) { - const manifest = { getMany: true } - AbstractLevelDOWN.call(this, manifest) - - this.jsonDB = jsonDB - this.wkbDB = wkbDB -} - -util.inherits(PartitionDOWN, AbstractLevelDOWN) - -const isGeometry = value => { - if (!value) return false - else if (typeof value !== 'object') return false - else { - if (!value.type) return false - else if (!value.coordinates && !value.geometries) return false - return true - } -} - -const safeget = async (level, key) => { - try { - return await level.get(key) - } catch (err) { - return undefined - } -} - -const safedel = async (level, key) => { - try { - return await level.del(key) - } catch (err) { - // Let it slide. - } -} - -/** - * _put :: k -> {k, v} - * _put :: k -> GeoJSON/Geometry - * _put :: k -> * - */ -PartitionDOWN.prototype._put = async function (key, value, options, callback) { - const err = this._checkKey(key) || this._checkValue(value) - if (err) return this._nextTick(callback, err) - - // Cases - // 1. value is GeoJSON/Geometry - // 2. value is object with geometry property - // 3. none of the above - - try { - if (isGeometry(value)) { - // 1. Only write geometry: - await this.wkbDB.put(key, value) - } else { - const { geometry, ...others } = value - // 2. Write geometry and other properties: - if (isGeometry(geometry)) { - await this.wkbDB.put(key, geometry) - await this.jsonDB.put(key, others) - } else { - // 3. Write value as-is: - await this.jsonDB.put(key, value) - } - } - - this._nextTick(callback) - } catch (err) { - this._nextTick(callback, err) - } -} - -/** - * _get :: k - */ -PartitionDOWN.prototype._get = async function (key, options, callback) { - const err = this._checkKey(key) - if (err) return this._nextTick(callback, err) - - try { - const geometry = await safeget(this.wkbDB, key) - const others = await safeget(this.jsonDB, key) - - if (isGeometry(geometry)) { - if (others === undefined) return this._nextTick(callback, null, geometry) - else return this._nextTick(callback, null, { geometry, ...others }) - } else { - if (others === undefined) return this._nextTick(callback, new Error('NotFound')) - else return this._nextTick(callback, null, others) - } - } catch (err) { - this._nextTick(callback, err) - } -} - -/** - * _getMany :: [k] - */ -PartitionDOWN.prototype._getMany = async function (keys, options, callback) { - const err = keys - .map(key => this._checkKey(key)) - .find(err => err) - - if (err) return this._nextTick(callback, err) - - try { - const geometry = await this.wkbDB.getMany(keys) - const others = await this.jsonDB.getMany(keys) - const entries = keys.map((_, index) => { - if (isGeometry(geometry[index])) { - if (!others[index]) return geometry[index] - else return { geometry: geometry[index], ...others[index] } - } else { - if (!others) return undefined - else return others[index] - } - }) - - return this._nextTick(callback, null, entries) - } catch (err) { - this._nextTick(callback, err) - } -} - -/** - * Note: We do not check if key exists at all. - */ -PartitionDOWN.prototype._del = async function (key, options, callback) { - const err = this._checkKey(key) - if (err) return this._nextTick(callback, err) - - try { - await safedel(this.wkbDB, key) - await safedel(this.jsonDB, key) - this._nextTick(callback) - } catch (err) { - this._nextTick(callback, err) - } -} - -/** - * - */ -PartitionDOWN.prototype._batch = async function (array, options, callback) { - - if (!Array.isArray(array)) { - return this._nextTick(callback, new Error('batch(array) requires an array argument')) - } - - const [geometries, properties] = array.reduce((acc, op) => { - const [geometries, properties] = acc - const { type, key, value } = op - - if (type === 'del') { - // For 'del' batch seems to ignore keys which do not exist. - geometries.push(op) - properties.push(op) - } else if (type === 'put') { - if (isGeometry(value)) geometries.push(op) - else { - const { geometry, ...others } = value - if (isGeometry(geometry)) { - geometries.push({ type: 'put', key, value: geometry }) - properties.push({ type: 'put', key, value: others }) - } else { - properties.push({ type: 'put', key, value }) - } - } - } - - return acc - }, [[], []]) - - try { - if (geometries.length) await this.wkbDB.batch(geometries) - if (properties.length) await this.jsonDB.batch(properties) - this._nextTick(callback) - } catch (err) { - this._nextTick(callback, err) - } -} - -/** - * - */ -PartitionDOWN.prototype._iterator = function (options) { - // Keys are necessary to synchronize iterators: - return new Iterator(this, { - ...options, - properties: this.jsonDB.iterator({ ...options, keys: true }), - geometry: this.wkbDB.iterator({ ...options, keys: true }) - }) -} diff --git a/src/shared/level/PartitionStore.js b/src/shared/level/PartitionStore.js new file mode 100644 index 00000000..7401dd51 --- /dev/null +++ b/src/shared/level/PartitionStore.js @@ -0,0 +1,176 @@ +/** + * Value-partitioning store. + * + * Routes a value's `geometry` to `wkbDB` (encoded as WKB) and the remaining + * properties to `jsonDB` (encoded as JSON). A plain adapter over two + * abstract-level child databases — it is not an abstract-level database + * itself, but exposes the subset of the API the `L.*` helpers and `Store` + * use: get / getMany / put / del / batch / iterator / keys / values. + */ +export function PartitionStore (jsonDB, wkbDB) { + this.jsonDB = jsonDB + this.wkbDB = wkbDB +} + +const isGeometry = value => { + if (!value) return false + if (typeof value !== 'object') return false + if (!value.type) return false + if (!value.coordinates && !value.geometries) return false + return true +} + +/** + * split :: value -> { geometry, properties } + * Either part may be `undefined`: + * - value is a geometry -> { geometry: value, properties: undefined } + * - value has a geometry property -> { geometry, properties: rest } + * - anything else -> { geometry: undefined, properties: value } + */ +const split = value => { + if (isGeometry(value)) return { geometry: value, properties: undefined } + + if (value && typeof value === 'object') { + const { geometry, ...properties } = value + if (isGeometry(geometry)) return { geometry, properties } + } + + return { geometry: undefined, properties: value } +} + +const checkKey = key => { + if (key === null || key === undefined) { + throw new Error('key cannot be `null` or `undefined`') + } +} + +const checkValue = value => { + if (value === null || value === undefined) { + throw new Error('value cannot be `null` or `undefined`') + } +} + +/** combine :: (geometry, properties) -> value */ +const combine = (geometry, properties) => { + if (isGeometry(geometry)) { + return properties === undefined ? geometry : { geometry, ...properties } + } + return properties +} + +PartitionStore.prototype.put = async function (key, value) { + checkKey(key) + checkValue(value) + const { geometry, properties } = split(value) + await Promise.all([ + geometry !== undefined ? this.wkbDB.put(key, geometry) : Promise.resolve(), + properties !== undefined ? this.jsonDB.put(key, properties) : Promise.resolve() + ]) +} + +PartitionStore.prototype.get = async function (key) { + checkKey(key) + const [geometry, properties] = await Promise.all([ + this.wkbDB.get(key), + this.jsonDB.get(key) + ]) + return combine(geometry, properties) +} + +PartitionStore.prototype.getMany = async function (keys) { + const [geometries, properties] = await Promise.all([ + this.wkbDB.getMany(keys), + this.jsonDB.getMany(keys) + ]) + return keys.map((_, index) => combine(geometries[index], properties[index])) +} + +PartitionStore.prototype.del = async function (key) { + checkKey(key) + await Promise.all([this.wkbDB.del(key), this.jsonDB.del(key)]) +} + +PartitionStore.prototype.batch = async function (operations) { + if (!Array.isArray(operations)) { + throw new Error('batch(array) requires an array argument') + } + + const geometryOps = [] + const propertyOps = [] + + for (const op of operations) { + checkKey(op.key) + if (op.type === 'del') { + // A partition that never held the key ignores the del. + geometryOps.push(op) + propertyOps.push(op) + } else if (op.type === 'put') { + checkValue(op.value) + const { geometry, properties } = split(op.value) + if (geometry !== undefined) geometryOps.push({ type: 'put', key: op.key, value: geometry }) + if (properties !== undefined) propertyOps.push({ type: 'put', key: op.key, value: properties }) + } + } + + await Promise.all([ + geometryOps.length ? this.wkbDB.batch(geometryOps) : Promise.resolve(), + propertyOps.length ? this.jsonDB.batch(propertyOps) : Promise.resolve() + ]) +} + +/** + * Merge the two child iterators (both ascending by key) into a single + * stream of reconstructed [key, value] entries. + */ +async function * merge (jsonDB, wkbDB, options) { + const opts = { ...options } + const limit = typeof opts.limit === 'number' && opts.limit >= 0 ? opts.limit : Infinity + delete opts.limit + + const properties = jsonDB.iterator(opts) + const geometries = wkbDB.iterator(opts) + + try { + let p = await properties.next() + let g = await geometries.next() + let count = 0 + + while (count < limit && (p !== undefined || g !== undefined)) { + let key, value + + if (g === undefined || (p !== undefined && p[0] < g[0])) { + // properties-only key + [key, value] = p + p = await properties.next() + } else if (p === undefined || g[0] < p[0]) { + // geometry-only key + [key, value] = g + g = await geometries.next() + } else { + // same key in both partitions + key = p[0] + value = combine(g[1], p[1]) + p = await properties.next() + g = await geometries.next() + } + + count += 1 + yield [key, value] + } + } finally { + await properties.close() + await geometries.close() + } +} + +PartitionStore.prototype.iterator = function (options) { + return merge(this.jsonDB, this.wkbDB, options || {}) +} + +PartitionStore.prototype.keys = async function * (options) { + for await (const [key] of this.iterator(options)) yield key +} + +PartitionStore.prototype.values = async function * (options) { + for await (const entry of this.iterator(options)) yield entry[1] +} diff --git a/src/shared/level/index.js b/src/shared/level/index.js index 511b8712..ee61a2f6 100644 --- a/src/shared/level/index.js +++ b/src/shared/level/index.js @@ -1,25 +1,26 @@ import * as R from 'ramda' -import levelup from 'levelup' -import leveldown from 'leveldown' -import memdown from 'memdown' -import sublevel from 'subleveldown' -import encode from 'encoding-down' +import { ClassicLevel } from 'classic-level' +import { MemoryLevel } from 'memory-level' import { wkb } from './wkb' -const encodings = { +// Value encodings accepted via the `encoding` factory option. +const valueEncodings = { wkb, - json: { valueEncoding: 'json' } + json: 'json' } +/** + * leveldb :: Options -> AbstractLevel + */ export const leveldb = (options = {}) => { - const encoding = encodings[options.encoding] - if (options.down) return levelup(options.down) - else if (options.up) return sublevel(options.up, options.prefix, encoding) - else { - const down = options.location ? leveldown(options.location) : memdown() - const encoded = encoding ? encode(down, encoding) : down - return leveldb({ down: encoded }) - } + const valueEncoding = valueEncodings[options.encoding] + const dbOptions = valueEncoding ? { valueEncoding } : {} + + if (options.parent) return options.parent.sublevel(options.prefix, dbOptions) + + return options.location + ? new ClassicLevel(options.location, dbOptions) + : new MemoryLevel(dbOptions) } @@ -27,28 +28,28 @@ export const leveldb = (options = {}) => { * JSON-encoded 'tuples' partition on top of plain store. * @param {*} db plain store without explicit encoding. */ -export const jsonDB = db => leveldb({ up: db, encoding: 'json', prefix: 'tuples' }) +export const jsonDB = db => leveldb({ parent: db, encoding: 'json', prefix: 'tuples' }) /** * WKB-encoded 'geometries' partition on top of plain store. * @param {*} db plain store without explicit encoding. */ -export const wkbDB = db => leveldb({ up: db, encoding: 'wkb', prefix: 'geometries' }) +export const wkbDB = db => leveldb({ parent: db, encoding: 'wkb', prefix: 'geometries' }) /** * JSON-encoded 'preferences' partition on top of plain store. * @param {*} db plain store without explicit encoding. */ -export const preferencesDB = db => sublevel(db, 'preferences', { valueEncoding: 'json' }) +export const preferencesDB = db => db.sublevel('preferences', { valueEncoding: 'json' }) /** * JSON-encoded 'session' partition on top of plain store. * @param {*} db plain store without explicit encoding. */ -export const sessionDB = db => sublevel(db, 'session', { valueEncoding: 'json' }) +export const sessionDB = db => db.sublevel('session', { valueEncoding: 'json' }) /** @@ -56,7 +57,7 @@ export const sessionDB = db => sublevel(db, 'session', { valueEncoding: 'json' } * Holds database schema options for upgrading/downgrading schema between versions. * @param {*} db plain store without explicit encoding. */ -export const schemaDB = db => sublevel(db, 'schema', { valueEncoding: 'json' }) +export const schemaDB = db => db.sublevel('schema', { valueEncoding: 'json' }) /** @@ -75,36 +76,26 @@ export const putOp = (key, value) => ({ type: 'put', key, value }) export const delOp = key => ({ type: 'del', key }) /** - * read :: (stream, fn) -> [fn(k, v)] + * collect :: (AsyncIterable a, a -> b) -> Promise [b] */ -export const read = (stream, decode) => new Promise((resolve, reject) => { +const collect = async (iterable, decode) => { const acc = [] - stream - .on('data', data => acc.push(decode(data))) - .on('error', reject) - .on('close', () => resolve(acc)) -}) - -export const Decoders = { - TUPLE: ({ key, value }) => [key, value], - ENTITY: ({ key, value }) => ({ id: key, ...value }) + for await (const item of iterable) acc.push(decode(item)) + return acc } -export const readStream = (db, options) => db.createReadStream(options) - -export const Streams = { - TUPLE: (db, options) => readStream(db, { ...options, keys: true, values: true }), - VALUE: (db, options) => readStream(db, { ...options, keys: false, values: true }), - KEY: (db, options) => readStream(db, { ...options, keys: true, values: false }) +export const Decoders = { + TUPLE: ([key, value]) => [key, value], + ENTITY: ([key, value]) => ({ id: key, ...value }) } -export const readTuples = (db, options) => read(Streams.TUPLE(db, options), Decoders.TUPLE) -export const readEntities = (db, options) => read(Streams.TUPLE(db, options), Decoders.ENTITY) -export const readKeys = (db, options) => read(Streams.KEY(db, options), R.identity) -export const readValues = (db, options) => read(Streams.VALUE(db, options), R.identity) +export const readTuples = (db, options) => collect(db.iterator(options), Decoders.TUPLE) +export const readEntities = (db, options) => collect(db.iterator(options), Decoders.ENTITY) +export const readKeys = (db, options) => collect(db.keys(options), R.identity) +export const readValues = (db, options) => collect(db.values(options), R.identity) /** - * mget :: fn -> (levelup, [k]) -> [fn(k, v)] + * mget :: fn -> (db, [k]) -> [fn(k, v)] */ export const mget = (decode, defaultValue) => async (db, keys) => { const values = await db.getMany(keys) @@ -121,22 +112,22 @@ export const mget = (decode, defaultValue) => async (db, keys) => { } /** - * mgetTuples :: (levelup, [k]) -> [[k, v]] + * mgetTuples :: (db, [k]) -> [[k, v]] */ export const mgetTuples = mget((key, value) => [key, value]) /** - * mgetKeys :: (levelup, [k]) -> [k] + * mgetKeys :: (db, [k]) -> [k] */ export const mgetKeys = mget((key, _) => key) /** - * mgetKeys :: (levelup, [k]) -> [v] + * mgetValues :: (db, [k]) -> [v] */ export const mgetValues = defaultValue => mget((_, value) => value, defaultValue) /** - * mgetEntities :: (levelup, [k]) -> [{id: k, ...v}] + * mgetEntities :: (db, [k]) -> [{id: k, ...v}] */ export const mgetEntities = mget((key, value) => ({ id: key, ...value })) @@ -161,42 +152,46 @@ export const keys = (db, arg) => Array.isArray(arg) : readKeys(db, {}) /** - * values :: levelup -> [k] -> [v] - * values :: levelup -> String -> [v] + * values :: db -> [k] -> [v] + * values :: db -> String -> [v] */ export const values = (db, arg, defaultValue) => Array.isArray(arg) ? mgetValues(defaultValue)(db, arg) : readValues(db, prefix(arg)) /** - * existsKey :: levelup -> String -> Boolean + * existsKey :: db -> {gte, lte} -> Boolean */ -export const existsKey = (db, prefix) => new Promise((resolve, reject) => { - db.createReadStream({ keys: true, values: false, limit: 1, ...prefix }) - .on('data', () => resolve(true)) - .on('error', reject) - .on('close', () => resolve(false)) -}) +export const existsKey = async (db, range) => { + const found = await collect(db.keys({ ...range, limit: 1 }), R.identity) + return found.length > 0 +} /** - * get :: levelup -> k -> v - * get :: levelup -> k -> v -> v + * get :: db -> k -> v + * get :: db -> k -> v -> v * * Get value for given key with optional default value if key was not found. + * abstract-level returns `undefined` for a missing key; PartitionStore and the + * IPC client may reject — both are treated as "not found". */ export const get = async (db, key, value) => { + let result try { - return await db.get(key) + result = await db.get(key) } catch (err) { - if (typeof value === 'undefined') throw err - else return value + result = undefined } + + if (result !== undefined) return result + if (typeof value === 'undefined') throw new Error(`key not found: ${key}`) + return value } /** - * mput :: levelup -> (k, v) -> unit - * mput :: levelup -> {k: v} -> unit - * mput :: levelup -> [[k, v]] -> unit + * mput :: db -> (k, v) -> unit + * mput :: db -> {k: v} -> unit + * mput :: db -> [[k, v]] -> unit */ export const mput = (db, ...args) => { if (args.length === 2) return db.put(args[0], args[1]) // key/value @@ -211,7 +206,7 @@ export const mput = (db, ...args) => { } /** - * mdel :: levelup -> [k] -> unit + * mdel :: db -> [k] -> unit */ export const mdel = async (db, arg) => { if (Array.isArray(arg)) return db.batch(arg.map(key => delOp(key))) @@ -219,7 +214,7 @@ export const mdel = async (db, arg) => { } /** - * tap :: levelup -> k -> (v -> v) -> unit + * tap :: db -> k -> (v -> v) -> unit */ export const tap = async function (db, key, fn) { const value = await db.get(key) diff --git a/src/shared/level/ipc.js b/src/shared/level/ipc.js index c742d72a..34b1af72 100644 --- a/src/shared/level/ipc.js +++ b/src/shared/level/ipc.js @@ -1,99 +1,64 @@ -import util from 'util' -import { AbstractLevelDOWN, AbstractIterator } from 'abstract-leveldown' - export const GET = 'level:get' export const PUT = 'level:put' export const DEL = 'level:del' export const ITERATOR = 'level:iterator' /** - * Iterator which fetches complete result at once. + * Level client which proxies operations to an `IPCServer` over IPC. * - * @param {*} db IPCDownClient instance - * @param {*} options iterator options - */ -function IPCIterator (db, options) { - AbstractIterator.call(this, db) - this._index = -1 - - // Promise of array which is comsumed in _next() - this._acc = db._ipc.invoke(ITERATOR, options) -} - -util.inherits(IPCIterator, AbstractIterator) - -IPCIterator.prototype._next = function (callback) { - this._acc.then(result => { - this._index += 1 - if (this._index === result.length) return this._nextTick(callback) - const { key, value } = result[this._index] - this._nextTick(callback, null, key, value) - }) -} - - -/** - * Leveldown implementation which communicates with server through IPC. + * A plain adapter exposing the subset of the API the `L.*` helpers use: + * get / put / del / iterator / keys / values. * * @param {*} ipc ipcMain or ipcRenderer instance. */ -export function IPCDownClient (ipc) { - AbstractLevelDOWN.call(this) +export function IPCClient (ipc) { this._ipc = ipc } -util.inherits(IPCDownClient, AbstractLevelDOWN) +IPCClient.prototype.get = function (key, options) { + return this._ipc.invoke(GET, key, options) +} + +IPCClient.prototype.put = function (key, value, options) { + return this._ipc.invoke(PUT, key, value, options) +} -IPCDownClient.prototype._get = function (key, options, callback) { - this._ipc.invoke(GET, key, options) - .then(value => this._nextTick(callback, null, value)) - .catch(err => this._nextTick(callback, err)) +IPCClient.prototype.del = function (key, options) { + return this._ipc.invoke(DEL, key, options) } -IPCDownClient.prototype._put = function (key, value, options, callback) { - this._ipc.invoke(PUT, key, value, options) - .then(() => this._nextTick(callback)) - .catch(err => this._nextTick(callback, err)) +/** + * Fetches the complete result at once, then yields it. The IPC round-trip + * does not support incremental streaming. + */ +IPCClient.prototype.iterator = async function * (options) { + const result = await this._ipc.invoke(ITERATOR, options) + for (const { key, value } of result) yield [key, value] } -IPCDownClient.prototype._del = function (key, options, callback) { - this._ipc.invoke(DEL, key, options) - .then(() => this._nextTick(callback)) - .catch(err => this._nextTick(callback, err)) +IPCClient.prototype.keys = async function * (options) { + for await (const [key] of this.iterator(options)) yield key } -IPCDownClient.prototype._iterator = function (options) { - return new IPCIterator(this, options) +IPCClient.prototype.values = async function * (options) { + for await (const entry of this.iterator(options)) yield entry[1] } /** - * Wrap Levelup database as IPC endpoint for IPC client. + * Exposes an abstract-level database as an IPC endpoint for `IPCClient`. * - * @param {*} db Levelup instance + * @param {*} db abstract-level database. * @param {*} ipc ipcMain instance. */ export function IPCServer (db, ipc) { - - ipc.handle(GET, async (event, key, options) => { - return await db.get(key) - }) - - ipc.handle(PUT, async (event, key, value, options) => { - return await db.put(key, value) - }) - - ipc.handle(DEL, async (event, key, options) => { - return await db.del(key) - }) + ipc.handle(GET, async (event, key, options) => db.get(key)) + ipc.handle(PUT, async (event, key, value, options) => db.put(key, value)) + ipc.handle(DEL, async (event, key, options) => db.del(key)) ipc.handle(ITERATOR, async (event, options) => { - return await new Promise((resolve, reject) => { - const acc = [] - db.createReadStream(options) - .on('data', data => acc.push(data)) - .on('error', err => reject(err)) - .on('end', () => resolve(acc)) - }) + const acc = [] + for await (const [key, value] of db.iterator(options)) acc.push({ key, value }) + return acc }) } diff --git a/src/shared/level/wkb.js b/src/shared/level/wkb.js index b85d154c..9e3fa05f 100644 --- a/src/shared/level/wkb.js +++ b/src/shared/level/wkb.js @@ -6,18 +6,17 @@ const toGeoJSON = geometry => geometry.toGeoJSON() const parse = wkx.Geometry.parse const toWkb = geometry => geometry.toWkb() +/** + * abstract-level custom value encoding: GeoJSON geometry <-> WKB buffer. + * Shape follows level-transcoder ({ name, format, encode, decode }). + */ export const wkb = { - valueEncoding: { - buffer: true, + name: 'wkb', + format: 'buffer', - /** - * Encode JSON (GeoJSON geometry) as WKB buffer. - */ - encode: R.compose(toWkb, parseGeoJSON), + /** Encode JSON (GeoJSON geometry) as WKB buffer. */ + encode: R.compose(toWkb, parseGeoJSON), - /** - * Deocde WKB buffer to JSON (GeoJSON geometry). - */ - decode: R.compose(toGeoJSON, parse) - } + /** Decode WKB buffer to JSON (GeoJSON geometry). */ + decode: R.compose(toGeoJSON, parse) } diff --git a/test/shared/level/PartitionDOWN-test.js b/test/shared/level/PartitionDOWN-test.js deleted file mode 100644 index f7335190..00000000 --- a/test/shared/level/PartitionDOWN-test.js +++ /dev/null @@ -1,296 +0,0 @@ -import assert from 'assert' -import * as R from 'ramda' -import { PartitionDOWN } from '../../../src/shared/level/PartitionDOWN' -import { leveldb, jsonDB, wkbDB } from '../../../src/shared/level' - -const geometry = { - geometry: { type: 'Point', coordinates: [1742867.2027975845, 5905160.9281057175] } -} - -const properties = { - type: 'Feature', - name: 'PzGrenKp Lipsch', - properties: { sidc: 'SHGPUCIZ--*E***', f: '(+)', n: 'ENY' } -} - -const fixture = [ - [{}, 'empty'], - [0, 'Number (0)'], - [1, 'Number (1)'], - ['', 'String ("")'], - ['XYZ', 'String ("XYZ")'], - [{ ...properties }, 'properties only'], - [{ ...geometry }, 'geometry only'], - [{ ...properties, ...geometry }, 'properties/geometry'] -] - -describe('PartitionDOWN', function () { - - const createdb = () => { - const db = leveldb({}) - const propertiesLevel = jsonDB(db) - const geometriesLevel = wkbDB(db) - const down = new PartitionDOWN(propertiesLevel, geometriesLevel) - return leveldb({ down }) - } - - it('get - key cannot be `null`', async function () { - try { - await createdb().get(null) - } catch (err) { - const expected = 'key cannot be `null` or `undefined`' - assert.strictEqual(expected, err.message) - } - }) - - it('get - key cannot be `undefined`', async function () { - try { - await createdb().get(undefined) - } catch (err) { - const expected = 'key cannot be `null` or `undefined`' - assert.strictEqual(expected, err.message) - } - }) - - it('get - key not found in database', async function () { - try { - await createdb().get('key') - } catch (err) { - const expected = 'Key not found in database [key]' - assert.strictEqual(expected, err.message) - } - }) - - it('put - key cannot be `null`', async function () { - try { - await createdb().put(null, 'value') - } catch (err) { - const expected = 'key cannot be `null` or `undefined`' - assert.strictEqual(expected, err.message) - } - }) - - it('put - key cannot be `undefined`', async function () { - try { - await createdb().put(undefined, 'value') - } catch (err) { - const expected = 'key cannot be `null` or `undefined`' - assert.strictEqual(expected, err.message) - } - }) - - it('put - value cannot be `null`', async function () { - try { - await createdb().put('key', null) - } catch (err) { - const expected = 'value cannot be `null` or `undefined`' - assert.strictEqual(expected, err.message) - } - }) - - it('put - value cannot be `undefined`', async function () { - try { - await createdb().put('key', undefined) - } catch (err) { - const expected = 'value cannot be `null` or `undefined`' - assert.strictEqual(expected, err.message) - } - }) - - it('put/get - value :: string', async function () { - const db = createdb() - const expected = 'value' - await db.put('key', expected) - const actual = await db.get('key') - assert.deepStrictEqual(actual, expected) - }) - - it('put/get - geometry', async function () { - const db = createdb() - const expected = { type: 'Point', coordinates: [1742867.2027975845, 5905160.9281057175] } - await db.put('key', expected) - const actual = await db.get('key') - assert.deepStrictEqual(actual, expected) - }) - - it('put/get - w/o geometry property', async function () { - const db = createdb() - const expected = { - type: 'Feature', - name: 'PzGrenKp Lipsch', - properties: { sidc: 'SHGPUCIZ--*E***', f: '(+)', n: 'ENY' } - } - - await db.put('key', expected) - const actual = await db.get('key') - assert.deepStrictEqual(actual, expected) - }) - - it('put/get - w/ geometry property', async function () { - const db = createdb() - const expected = { - type: 'Feature', - name: 'PzGrenKp Lipsch', - geometry: { type: 'Point', coordinates: [1742867.2027975845, 5905160.9281057175] }, - properties: { sidc: 'SHGPUCIZ--*E***', f: '(+)', n: 'ENY' } - } - - await db.put('key', expected) - const actual = await db.get('key') - assert.deepStrictEqual(actual, expected) - }) - - it('del - key cannot be `null`', async function () { - try { - await createdb().del(null) - } catch (err) { - const expected = 'key cannot be `null` or `undefined`' - assert.strictEqual(expected, err.message) - } - }) - - it('del - key cannot be `undefined`', async function () { - try { - await createdb().del(undefined) - } catch (err) { - const expected = 'key cannot be `null` or `undefined`' - assert.strictEqual(expected, err.message) - } - }) - - it('put/del - value :: string', async function () { - const db = createdb() - const expected = 'value' - await db.put('key', expected) - await db.del('key') - - try { - await db.get('key') - } catch (err) { - const expected = 'Key not found in database [key]' - assert.deepEqual(err.message, expected) - } - }) - - it('put/del - geometry', async function () { - const db = createdb() - const expected = { type: 'Point', coordinates: [1742867.2027975845, 5905160.9281057175] } - await db.put('key', expected) - await db.del('key') - - try { - await db.get('key') - } catch (err) { - const expected = 'Key not found in database [key]' - assert.deepEqual(err.message, expected) - } - }) - - it('put/del - w/o geometry property', async function () { - const db = createdb() - const expected = { - type: 'Feature', - name: 'PzGrenKp Lipsch', - properties: { sidc: 'SHGPUCIZ--*E***', f: '(+)', n: 'ENY' } - } - - await db.put('key', expected) - await db.del('key') - - try { - await db.get('key') - } catch (err) { - const expected = 'Key not found in database [key]' - assert.deepEqual(err.message, expected) - } - }) - - it('put/del - w/ geometry property', async function () { - const db = createdb() - const expected = { - type: 'Feature', - name: 'PzGrenKp Lipsch', - geometry: { type: 'Point', coordinates: [1742867.2027975845, 5905160.9281057175] }, - properties: { sidc: 'SHGPUCIZ--*E***', f: '(+)', n: 'ENY' } - } - - await db.put('key', expected) - await db.del('key') - - try { - await db.get('key') - } catch (err) { - const expected = 'Key not found in database [key]' - assert.deepEqual(err.message, expected) - } - }) - - ;[ - [undefined, 'undefined'], - ['', 'String'], - [0, 'Number'], - [{}, 'Object'], - ].forEach(([value, type]) => { - it(`batch - requires an array argument [${type}]`, async function () { - try { - const db = createdb() - await db.batch(value) - } catch (actual) { - assert.strictEqual(actual.message, 'batch(array) requires an array argument') - } - }) - }) - - fixture.forEach(([expected, description]) => { - it(`batch/put [${description}]`, async function () { - const db = createdb() - await db.batch([{ type: 'put', key: 'key', value: expected }]) - const actual = await db.get('key') - assert.deepStrictEqual(actual, expected) - }) - }) - - fixture.forEach(([expected, description]) => { - it(`batch/del [${description}]`, async function () { - const db = createdb() - await db.batch([{ type: 'put', key: 'key', value: expected }]) - await db.batch([{ type: 'del', key: 'key' }]) - const actual = await db.getMany(['key']) - assert.deepStrictEqual(actual, [undefined]) - }) - }) - - const list = (db, options) => new Promise((resolve, reject) => { - const acc = [] - db.createReadStream(options) - .on('data', data => acc.push(data)) - .on('err', reject) - .on('close', () => resolve(acc)) - }) - - describe('createReadStream', function () { - it('{ keys: true, value: true }', async function () { - const db = createdb() - const expected = fixture.map(([value], i) => ({ key: String(100 + i), value })) - await db.batch(expected.map(kv => ({ type: 'put', ...kv }))) - const actual = await list(db, {}) - assert.deepStrictEqual(actual, expected) - }) - - it('{ keys: false, value: true }', async function () { - const db = createdb() - const expected = fixture.map(([value], i) => ({ key: String(100 + i), value })) - await db.batch(expected.map(kv => ({ type: 'put', ...kv }))) - const actual = await list(db, { keys: false }) - assert.deepStrictEqual(actual, expected.map(R.prop('value'))) - }) - - it('{ keys: true, value: false }', async function () { - const db = createdb() - const expected = fixture.map(([value], i) => ({ key: String(100 + i), value })) - await db.batch(expected.map(kv => ({ type: 'put', ...kv }))) - const actual = await list(db, { values: false }) - assert.deepStrictEqual(actual, expected.map(R.prop('key'))) - }) - }) -}) diff --git a/test/shared/level/PartitionStore-test.js b/test/shared/level/PartitionStore-test.js new file mode 100644 index 00000000..edfbc8ec --- /dev/null +++ b/test/shared/level/PartitionStore-test.js @@ -0,0 +1,104 @@ +import assert from 'assert' +import { PartitionStore } from '../../../src/shared/level/PartitionStore' +import { leveldb, jsonDB, wkbDB } from '../../../src/shared/level' + +const createdb = () => { + const db = leveldb({}) + return new PartitionStore(jsonDB(db), wkbDB(db)) +} + +// One sample per routing case: properties-only, geometry-only and split. +const samples = [ + ['a string', 'value'], + ['a number', 0], + ['a geometry', { type: 'Point', coordinates: [1742867.2027975845, 5905160.9281057175] }], + ['a feature without geometry', { + type: 'Feature', + name: 'PzGrenKp Lipsch', + properties: { sidc: 'SHGPUCIZ--*E***', f: '(+)', n: 'ENY' } + }], + ['a feature with geometry', { + type: 'Feature', + name: 'PzGrenKp Lipsch', + geometry: { type: 'Point', coordinates: [1742867.2027975845, 5905160.9281057175] }, + properties: { sidc: 'SHGPUCIZ--*E***', f: '(+)', n: 'ENY' } + }] +] + +describe('PartitionStore', function () { + + describe('argument validation', function () { + it('get rejects a null/undefined key', async function () { + await assert.rejects(() => createdb().get(null), /key cannot be/) + await assert.rejects(() => createdb().get(undefined), /key cannot be/) + }) + + it('put rejects a null/undefined key', async function () { + await assert.rejects(() => createdb().put(null, 'value'), /key cannot be/) + await assert.rejects(() => createdb().put(undefined, 'value'), /key cannot be/) + }) + + it('put rejects a null/undefined value', async function () { + await assert.rejects(() => createdb().put('key', null), /value cannot be/) + await assert.rejects(() => createdb().put('key', undefined), /value cannot be/) + }) + + it('del rejects a null/undefined key', async function () { + await assert.rejects(() => createdb().del(null), /key cannot be/) + await assert.rejects(() => createdb().del(undefined), /key cannot be/) + }) + + it('batch rejects a non-array argument', async function () { + for (const arg of [undefined, '', 0, {}]) { + await assert.rejects(() => createdb().batch(arg), /requires an array argument/) + } + }) + }) + + samples.forEach(([description, value]) => { + it(`put/get round-trips ${description}`, async function () { + const db = createdb() + await db.put('key', value) + assert.deepStrictEqual(await db.get('key'), value) + }) + + it(`put/del removes ${description}`, async function () { + const db = createdb() + await db.put('key', value) + await db.del('key') + assert.strictEqual(await db.get('key'), undefined) + }) + + it(`batch put/get round-trips ${description}`, async function () { + const db = createdb() + await db.batch([{ type: 'put', key: 'key', value }]) + assert.deepStrictEqual(await db.get('key'), value) + }) + + it(`batch del removes ${description}`, async function () { + const db = createdb() + await db.batch([{ type: 'put', key: 'key', value }]) + await db.batch([{ type: 'del', key: 'key' }]) + assert.deepStrictEqual(await db.getMany(['key']), [undefined]) + }) + }) + + it('iterator merges both partitions in key order', async function () { + const db = createdb() + const entries = samples.map(([, value], i) => [`key:${100 + i}`, value]) + await db.batch(entries.map(([key, value]) => ({ type: 'put', key, value }))) + + const acc = [] + for await (const entry of db.iterator({})) acc.push(entry) + assert.deepStrictEqual(acc, entries) + }) + + it('getMany reconstructs values for multiple keys', async function () { + const db = createdb() + const entries = samples.map(([, value], i) => [`key:${100 + i}`, value]) + await db.batch(entries.map(([key, value]) => ({ type: 'put', key, value }))) + + const actual = await db.getMany(entries.map(([key]) => key)) + assert.deepStrictEqual(actual, entries.map(([, value]) => value)) + }) +}) diff --git a/test/shared/level/ipc-test.js b/test/shared/level/ipc-test.js index 72abcc50..60412a40 100644 --- a/test/shared/level/ipc-test.js +++ b/test/shared/level/ipc-test.js @@ -1,79 +1,41 @@ import assert from 'assert' -import { IPCDownClient, IPCServer, GET, PUT, DEL, ITERATOR } from '../../../src/shared/level/ipc' +import { IPCClient, IPCServer, GET, PUT, DEL, ITERATOR } from '../../../src/shared/level/ipc' import { leveldb } from '../../../src/shared/level' -describe('IPCDownClient', function () { - it('GET', async function () { +describe('IPCClient', function () { + it('get', async function () { const values = { a: 0 } - const client = new IPCDownClient({ - invoke: async function (message, key, options) { - return values[key] - } - }) - - const db = leveldb({ down: client }) - const actual = await db.get('a') - assert.strictEqual(actual, 0) + const client = new IPCClient({ invoke: async (message, key) => values[key] }) + assert.strictEqual(await client.get('a'), 0) }) - it('GET (key not found)', async function () { - const client = new IPCDownClient({ - invoke: async function (message, key, options) { - throw new Error(`Key not found in database [${key}]`) - } + it('get rejects when the server rejects', async function () { + const client = new IPCClient({ + invoke: async (message, key) => { throw new Error(`key not found [${key}]`) } }) - - const db = leveldb({ down: client }) - try { - await db.get('a') - assert.fail() - } catch (err) { - // all good. - } + await assert.rejects(() => client.get('a')) }) - it('PUT', async function () { + it('put', async function () { const values = {} - const client = new IPCDownClient({ - invoke: async function (message, key, value, options) { - values[key] = value - } - }) - - const db = leveldb({ down: client }) - await db.put('a', 0) + const client = new IPCClient({ invoke: async (message, key, value) => { values[key] = value } }) + await client.put('a', 0) assert.strictEqual(values.a, 0) }) - it('DEL', async function () { + it('del', async function () { const values = { a: 0 } - const client = new IPCDownClient({ - invoke: async function (message, key, options) { - delete values[key] - } - }) - - const db = leveldb({ down: client }) - await db.del('a') + const client = new IPCClient({ invoke: async (message, key) => { delete values[key] } }) + await client.del('a') assert.deepStrictEqual(values, {}) }) - it('ITERATOR', async function () { + it('iterator', async function () { const expected = [{ key: 'a', value: 0 }, { key: 'b', value: 1 }] - const client = new IPCDownClient({ - invoke: async function (message, options) { - return expected - } - }) - - const db = leveldb({ down: client }) - const actual = await new Promise(resolve => { - const acc = [] - db.createReadStream() - .on('data', data => acc.push(data)) - .on('end', () => resolve(acc)) - }) + const client = new IPCClient({ invoke: async () => expected }) + const actual = [] + for await (const [key, value] of client.iterator()) actual.push({ key, value }) assert.deepStrictEqual(actual, expected) }) }) @@ -89,48 +51,31 @@ describe('IPCServer', function () { it('GET', async function () { const db = leveldb({ encoding: 'json' }) await db.put('a', 0) - /* eslint-disable no-new */ - new IPCServer(db, ipc) - /* eslint-enable no-new */ - const actual = await ipc.invoke(GET, 'a') - assert.strictEqual(actual, 0) + new IPCServer(db, ipc) // eslint-disable-line no-new + assert.strictEqual(await ipc.invoke(GET, 'a'), 0) }) it('PUT', async function () { const db = leveldb({ encoding: 'json' }) - /* eslint-disable no-new */ - new IPCServer(db, ipc) - /* eslint-enable no-new */ + new IPCServer(db, ipc) // eslint-disable-line no-new await ipc.invoke(PUT, 'a', 0) - const actual = await db.get('a') - assert.strictEqual(actual, 0) + assert.strictEqual(await db.get('a'), 0) }) it('DEL', async function () { const db = leveldb({ encoding: 'json' }) await db.put('a', 0) - /* eslint-disable no-new */ - new IPCServer(db, ipc) - /* eslint-enable no-new */ + new IPCServer(db, ipc) // eslint-disable-line no-new await ipc.invoke(DEL, 'a') - - try { - await db.get('a') - assert.fail() - } catch (err) { - // all good - } + assert.strictEqual(await db.get('a'), undefined) }) it('ITERATOR', async function () { const db = leveldb({ encoding: 'json' }) await db.put('a', 0) await db.put('b', 1) - /* eslint-disable no-new */ - new IPCServer(db, ipc) - /* eslint-enable no-new */ + new IPCServer(db, ipc) // eslint-disable-line no-new const actual = await ipc.invoke(ITERATOR, { keys: true, values: true }) - const expected = [{ key: 'a', value: 0 }, { key: 'b', value: 1 }] - assert.deepStrictEqual(actual, expected) + assert.deepStrictEqual(actual, [{ key: 'a', value: 0 }, { key: 'b', value: 1 }]) }) }) diff --git a/test/shared/level/wkb-test.js b/test/shared/level/wkb-test.js index 4a4526d3..587dfcd8 100644 --- a/test/shared/level/wkb-test.js +++ b/test/shared/level/wkb-test.js @@ -1,21 +1,96 @@ import assert from 'assert' +import * as L from '../../../src/shared/level' import { leveldb, wkbDB } from '../../../src/shared/level' +/** + * Regression coverage for the WKB value encoding. + * + * Exercises the WKB value encoding through the public surface (`leveldb`, + * `wkbDB`, `L.*` helpers). + */ describe('WKB encoding', function () { - it('encodes/decodes GeoJSON geometry as WKB', async function () { - const db = leveldb({ encoding: 'json' }) - const geometries = wkbDB(db) - const expected = { + const geometries = () => wkbDB(leveldb({ encoding: 'json' })) + + // One representative of each geometry type wkx round-trips through WKB. + const samples = { + Point: { + type: 'Point', + coordinates: [15.561677802092738, 46.82068398056285] + }, + LineString: { type: 'LineString', coordinates: [ [15.561677802092738, 46.82068398056285], [15.567283499146976, 46.81122129030928], [15.572291255182089, 46.79587284762624] ] + }, + Polygon: { + type: 'Polygon', + coordinates: [ + [[15.5, 46.8], [15.6, 46.8], [15.6, 46.9], [15.5, 46.9], [15.5, 46.8]], + [[15.52, 46.82], [15.55, 46.82], [15.55, 46.85], [15.52, 46.85], [15.52, 46.82]] + ] + }, + MultiPoint: { + type: 'MultiPoint', + coordinates: [[15.56, 46.82], [15.57, 46.81], [15.58, 46.80]] + }, + MultiLineString: { + type: 'MultiLineString', + coordinates: [ + [[15.56, 46.82], [15.57, 46.81]], + [[15.58, 46.80], [15.59, 46.79]] + ] + }, + MultiPolygon: { + type: 'MultiPolygon', + coordinates: [ + [[[15.5, 46.8], [15.6, 46.8], [15.6, 46.9], [15.5, 46.8]]], + [[[15.7, 46.7], [15.8, 46.7], [15.8, 46.8], [15.7, 46.7]]] + ] + }, + GeometryCollection: { + type: 'GeometryCollection', + geometries: [ + { type: 'Point', coordinates: [15.56, 46.82] }, + { type: 'LineString', coordinates: [[15.56, 46.82], [15.57, 46.81]] } + ] } + } + + Object.entries(samples).forEach(([name, geometry]) => { + it(`round-trips a ${name} through put/get`, async function () { + const db = geometries() + await db.put('key', geometry) + assert.deepStrictEqual(await db.get('key'), geometry) + }) + }) + + it('round-trips multiple geometries through batch and getMany', async function () { + const db = geometries() + const entries = Object.entries(samples) + await db.batch(entries.map(([key, value]) => L.putOp(key, value))) + + const keys = entries.map(([key]) => key) + const values = await db.getMany(keys) + assert.deepStrictEqual(values, entries.map(([, value]) => value)) + }) + + it('round-trips geometries through an iterator (readTuples)', async function () { + const db = geometries() + const entries = Object.entries(samples).sort(([a], [b]) => a < b ? -1 : 1) + await db.batch(entries.map(([key, value]) => L.putOp(key, value))) + + const tuples = await L.readTuples(db, {}) + assert.deepStrictEqual(tuples, entries) + }) - await geometries.put('key', expected) - const actual = await geometries.get('key') - assert.deepStrictEqual(actual, expected) + it('deletes a geometry', async function () { + const db = geometries() + await db.put('key', samples.Point) + await db.del('key') + // abstract-level returns undefined for a missing key (no rejection). + assert.strictEqual(await db.get('key'), undefined) }) }) diff --git a/webpack.config.js b/webpack.config.js index 6a31a7a5..cfb1d6b7 100644 --- a/webpack.config.js +++ b/webpack.config.js @@ -97,7 +97,9 @@ const rendererConfig = (env, argv) => ({ plugins: [ // Title is managed by BrowserWindow title option. new HtmlWebpackPlugin(), - new webpack.ExternalsPlugin('commonjs', ['leveldown']) + // classic-level is a native module — keep it as a runtime require so + // node-gyp-build resolves the prebuilt binary from node_modules. + new webpack.ExternalsPlugin('commonjs', ['classic-level']) ], externals: { // unused dependencies referenced by jsPDF @@ -117,7 +119,8 @@ const mainConfig = (env, argv) => ({ }, plugins: [ // NOTE: Required. Else "Error: No native build was found for ..." - new webpack.ExternalsPlugin('commonjs', ['leveldown']) + // classic-level is a native module and must not be bundled. + new webpack.ExternalsPlugin('commonjs', ['classic-level']) ], externals: { // unused dependencies referenced by jsPDF