diff --git a/AGENTS.md b/AGENTS.md index c08a9b3..f6f6169 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,13 +19,13 @@ The measurable result: 3–10× faster than SQLite on aggregate and scan workloa - **Embedded / edge / serverless workloads** where query latency is a tight budget and you don't want SQLite's quirks. - **Single-node analytics** over tables that fit on disk. The scan path is zero-syscall (mmap) and filters are compiled to byte-level predicates. - **You control both sides** (the DB and the app). PowDB has no Postgres wire protocol, no ODBC, no legacy compatibility. The client is a TCP binary protocol or an in-process Engine. -- **You want to read the code.** Four crates, ~20K lines, no generated parsers, no plan-language IR. +- **You want to read the code.** Eight crates, ~45K lines, no generated parsers, no plan-language IR. ### When it's *not* the right choice - You need a drop-in Postgres/MySQL replacement. PowQL is a different language; there is **no SQL compatibility layer**, and that is a deliberate design decision (the translation tier is the thing we're removing). - You need multi-node replication, sharding, or Raft-style consensus. Single-node only today. -- You need fine-grained ACLs, row-level security, or multi-tenant isolation. Auth is a single shared password. +- You need fine-grained ACLs, row-level security, or multi-tenant isolation. Auth is named users with coarse roles (admin/readwrite/readonly) since 0.4.5 — shared-password mode still available — but nothing per-table or per-row. - You need user-defined functions, stored procedures, or triggers. --- @@ -109,6 +109,7 @@ Compare SQL: `SELECT name, age FROM User WHERE age > 25 ORDER BY age DESC LIMIT | `AND`, `OR`, `NOT` | `and`, `or`, `not` (lowercase) | | `User.posts` (link navigation) | not yet implemented | | `let x := ...` | not yet implemented | +| `count: count(.name)` (aggregate keyword as alias) | fails `expected alias name` — `sum:` fails too; use `n:`, `cnt:`, `total:` | --- @@ -116,7 +117,7 @@ Compare SQL: `SELECT name, age FROM User WHERE age > 25 ORDER BY age DESC LIMIT Canonical type names: `str`, `int`, `float`, `bool`, `datetime`, `uuid`, `bytes`. -**Footgun:** the executor's type resolver falls back to `TypeId::Str` for any unknown name (`crates/query/src/executor.rs`), so `string`, `varchar`, or a typo silently produces a Str column with no error. Always use the canonical names above. +**Footgun:** the executor's type resolver falls back to `TypeId::Str` for any unknown name (`crates/query/src/executor/`), so `string`, `varchar`, or a typo silently produces a Str column with no error. Always use the canonical names above. `required` is a prefix keyword on the field, not a `!` suffix: `required name: str`, never `name: str!`. @@ -128,7 +129,7 @@ These are the design moves that buy the speedup. Understanding them keeps you fr 1. **Planner is a pure function.** It does not touch the catalog — it emits `RangeScan` speculatively, and the executor lowers to `Filter(SeqScan)` at runtime if no index exists. This keeps the parser → plan pipeline allocation-free for cache hits. 2. **Plan cache hashes canonical PowQL.** Literals are substituted at lookup time (FNV-1a hash, `crates/query/src/plan_cache.rs`). A repeated `User filter .id = ` reuses the same plan for all N. -3. **Compiled integer predicates.** `Filter(SeqScan)` on simple numeric predicates compiles into a branch-free byte-level check that skips full row decoding. See `execute_plan` fast paths in `crates/query/src/executor.rs`. +3. **Compiled integer predicates.** `Filter(SeqScan)` on simple numeric predicates compiles into a branch-free byte-level check that skips full row decoding. See `execute_plan` fast paths in `crates/query/src/executor/` (module dir). 4. **mmap-based scans.** The storage layer exposes `try_for_each_row_raw` over memory-mapped heap files. Early termination is a `return ControlFlow::Break`. 5. **Slotted 4KB pages + persistent B+tree indexes.** Standard, but the index format (BIDX, binary) is crash-safe and survives restart with no rebuild. 6. **WAL with group commit at statement boundaries.** Writes are durable by default; throughput is maintained by batching. @@ -157,6 +158,8 @@ cargo run --release -p powdb-cli # embedded REPL cargo run --release -p powdb-cli -- --remote host:5433 --password ``` +**The REPL is line-oriented.** A statement split across lines fails to parse — write each statement on one line. + ### TCP server ```bash @@ -201,9 +204,9 @@ Return shapes: ## What's shipped vs. what's planned -Shipped: joins (inner/left/right/cross, nested-loop + hash), GROUP BY + HAVING, DISTINCT, UNION / UNION ALL, subqueries (IN, EXISTS, correlated), CASE, LIKE, BETWEEN, IN-list, window functions (ROW_NUMBER, RANK, DENSE_RANK, SUM/AVG/COUNT/MIN/MAX over partition), arithmetic, string/math/datetime scalars, CAST, COALESCE, materialized views with auto-refresh, upsert, prepared queries with literal substitution, explicit transactions (`begin` / `commit` / `rollback`), password auth, TLS (`POWDB_TLS_CERT` / `POWDB_TLS_KEY`), WAL + crash recovery, persistent indexes. +Shipped: joins (inner/left/right/cross, nested-loop + hash), GROUP BY + HAVING, DISTINCT, UNION / UNION ALL, subqueries (IN, EXISTS, correlated), CASE, LIKE, BETWEEN, IN-list, window functions (ROW_NUMBER, RANK, DENSE_RANK, SUM/AVG/COUNT/MIN/MAX over partition), arithmetic, string/math/datetime scalars, CAST, COALESCE (`??`), materialized views with auto-refresh, upsert, multi-row INSERT, prepared queries with literal substitution, explicit transactions (`begin` / `commit` / `rollback`), password auth + multi-user auth (named users, admin/readwrite/readonly roles), TLS (`POWDB_TLS_CERT` / `POWDB_TLS_KEY`), WAL + crash recovery, persistent indexes, backup/restore (full/incremental/PITR, offline). -Planned (design doc only — don't use): link navigation (`User.posts`), `let` bindings, default operator (`??`), UDFs, per-row permissions, replication. +Planned (design doc only — don't use): link navigation (`User.posts`), `let` bindings, UDFs, per-row permissions, replication. --- @@ -211,9 +214,7 @@ Planned (design doc only — don't use): link navigation (`User.posts`), `let` b Build: `cargo build --workspace`. Test: `cargo test --workspace`. Lint: `cargo clippy --workspace --all-targets -- -D warnings`. Format: `cargo fmt --all`. -CI gates on `main`: -- `clippy + fmt + test` — `.github/workflows/ci.yml` -- `criterion + regression gate` — `.github/workflows/bench.yml` (blocks merges if any of 7 load-bearing workloads regress >7% against the checked-in baseline) +CI gates on `main` (all in `.github/workflows/ci.yml`): clippy/fmt/test (2-OS matrix), miri, asan, cargo audit, MSRV consistency, examples-smoke. `.github/workflows/bench.yml` (criterion + regression gate) is **manual-only** (`workflow_dispatch`) and NOT a merge gate — run the gate locally instead (see above). Internal docs: - `CLAUDE.md` — codebase guide for Claude Code (architecture, crate graph, common patterns) diff --git a/CHANGELOG.md b/CHANGELOG.md index e42097d..fe769ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed +- **Oversized rows no longer kill the server (remote DoS).** Inserting or + updating a row whose encoded size exceeds one 4 KB page (`MAX_ROW_DATA_SIZE`, + 4070 bytes) previously hit a `panic!` in the heap layer, which — combined with + `panic = "abort"` — terminated the entire server process. Any connected client + could take down the database with a single ~5 KB string insert. The heap now + rejects oversized rows with a graceful `row too large: N bytes exceeds max M + bytes` query error before anything is written or WAL-logged (an oversized + update can no longer poison WAL replay), and the connection and server keep + running. There is still no large-object/overflow-page support — values over + ~4 KB are rejected, not stored. +- **`readonly` role is now enforced at the query layer.** Previously the role + was authenticated and stored but never checked: a `readonly` user could + insert, update, delete, and drop tables. The server now classifies each + parsed statement and rejects writes (DML, DDL, view DDL, and transaction + control) from `readonly` principals with `permission denied: role 'readonly' + cannot execute write statements`. Unknown roles fail closed. Shared-password + mode, open mode, and embedded use are unaffected. +- **NULL values now arrive as `null` on the wire instead of `{}`.** The server + serialized SQL NULLs as `{}`, which the remote CLI displayed verbatim and + which broke the TS client's documented `"null"` sentinel for typed-row + decoding. The wire serialization is now the bareword `null`, and the remote + REPL renders it as `NULL`, matching embedded mode. +- **Window aggregates without `order` now compute the whole-partition value.** + `avg(.x) over (partition .d)` previously returned a running aggregate per + row (frame = partition start → current row) even with no `order` clause; per + standard semantics the frame is now the entire partition. Ordered windows + keep the running-frame behavior; ranking functions are unchanged. + +### Documentation +- Ecosystem-wide accuracy sweep: site pages synced to v0.4.5 (banners were + v0.2.0, MSRV corrected to 1.93), crates.io homepage URL fixed (was a 404), + README/CONTRIBUTING/AGENTS no longer claim the bench suite is a CI merge + gate, SECURITY.md documents both auth modes and the ≤0.4.5 readonly caveat, + RELEASES.md covers all six crates + the Docker image, deploy examples fixed + (`fly.toml` was missing `POWDB_BIND=0.0.0.0`), TS client docs document the + multi-user-server incompatibility, and AGENTS.md gained small-model-tested + gotchas (reserved aggregate keywords as aliases; line-oriented REPL). + ## [0.4.5] - 2026-06-09 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 2e73a51..f1699d4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,9 @@ PowDB is a from-scratch database engine with its own query language (PowQL). No powdb-cli ──→ powdb-server ──→ powdb-query ──→ powdb-storage ↑ ↑ powdb-bench powdb-compare + +powdb-auth ←── powdb-server, powdb-cli (user store + roles; no inter-crate deps) +powdb-backup ←── powdb-cli (depends on powdb-storage + powdb-query) ``` ### Query Pipeline @@ -30,7 +33,7 @@ PowQL text → Lexer (token stream) → Parser (AST) → Planner (PlanNode tree) - **Lexer** (`crates/query/src/lexer.rs`): Tokenizes PowQL input - **Parser** (`crates/query/src/parser.rs`): Recursive descent, produces `Statement` AST - **Planner** (`crates/query/src/planner.rs`): Pure function (no catalog access), produces `PlanNode` tree. Speculatively emits `RangeScan` for range inequalities -- **Executor** (`crates/query/src/executor.rs`): Runs plans against the storage engine. Has fast paths for common patterns (count, project+limit, sort+limit, agg, update, delete). Lowers `RangeScan` → `Filter(SeqScan)` at runtime when no index exists +- **Executor** (`crates/query/src/executor/` module dir): Runs plans against the storage engine. Has fast paths for common patterns (count, project+limit, sort+limit, agg, update, delete). Lowers `RangeScan` → `Filter(SeqScan)` at runtime when no index exists - **Plan Cache** (`crates/query/src/plan_cache.rs`): FNV-1a hash, stores canonical plans, substitutes literals at lookup time ### Storage Engine @@ -81,7 +84,7 @@ cargo run -p powdb-bench --bin compare 3. Add parser production to `crates/query/src/parser.rs` 4. Add plan node (if needed) to `crates/query/src/plan.rs` 5. Add planner case to `crates/query/src/planner.rs` -6. Add executor case to `crates/query/src/executor.rs` +6. Add executor case to `crates/query/src/executor/` (start in `mod.rs` / `plan_exec.rs`) ### Adding an executor fast path Fast paths match on specific `PlanNode` shapes in `execute_plan()`. Pattern-match the plan tree and handle it before the generic recursive executor. Always verify with benchmarks. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bca7ce7..bb12b3e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,6 +29,8 @@ cargo run --release -p powdb-compare # wide bench vs SQLite + Postgres (add --f ``` crates/storage # slotted pages, B+ tree, WAL, buffer pool, catalog crates/query # lexer, parser, planner, executor (Engine) +crates/auth # user store, roles, argon2id password hashing +crates/backup # offline backup/restore (full, incremental, PITR) crates/server # Tokio TCP server + binary wire protocol crates/cli # rustyline REPL (embedded + remote mode) crates/bench # criterion benchmarks + regression gate @@ -49,7 +51,7 @@ clients/ts # TypeScript client + demo ### Branch protection on `main` - PRs are required (no direct pushes) -- 7 status checks must pass: clippy + fmt + test (x2 OS matrix), miri, asan, audit, MSRV consistency, and the bench regression gate +- 7 status checks must pass, all from `ci.yml`: clippy + fmt + test (x2 OS matrix), miri, asan, audit, MSRV consistency, and examples-smoke - Force-push is rejected by branch protection Admin bypass exists for break-glass scenarios (security patches, recovering from a broken state). **Do not use it for routine work** — routine work goes through PRs even when bypass is technically available. @@ -74,11 +76,13 @@ PRs must pass these gates (see `.github/workflows/`): - **asan** — AddressSanitizer run - **audit** — `cargo audit` against the advisory database - **msrv-consistency** — verifies the declared MSRV (`1.93`) builds -- **criterion + regression gate** — benchmark must not regress beyond thresholds (`.github/workflows/bench.yml`) +- **examples-smoke** — terraform validate + compose config + dev.sh cycle on the deploy examples + +The criterion benchmark suite (`.github/workflows/bench.yml`) is **manual-only** (`workflow_dispatch`) and is *not* a required PR gate — shared-runner noise makes it unreliable as a blocking check. Run the regression gate locally instead (below). ## Benchmark Regression Gate -The criterion gate compares each workload's median against baselines in `crates/bench/baseline/main.json`. Thresholds vary by workload (7-20%). +The criterion gate compares each workload's median against baselines in `crates/bench/baseline/main.json`. Thresholds vary by workload (7-20%). Run it locally with `cargo bench -p powdb-bench && cargo run -p powdb-bench --bin compare`, or on demand in CI with `gh workflow run bench.yml`. If you intentionally change performance characteristics: ```bash diff --git a/Cargo.toml b/Cargo.toml index e406eed..fb8fb84 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -8,7 +8,7 @@ edition = "2021" rust-version = "1.93" license = "MIT" repository = "https://github.com/zvndev/powdb" -homepage = "https://zvndev.github.io/powdb/" +homepage = "https://zvn-dev.github.io/powdb/" readme = "README.md" [workspace.dependencies] diff --git a/README.md b/README.md index 2a6242b..f13c038 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ # PowDB [![CI](https://github.com/zvndev/powdb/actions/workflows/ci.yml/badge.svg)](https://github.com/zvndev/powdb/actions/workflows/ci.yml) -[![bench](https://github.com/zvndev/powdb/actions/workflows/bench.yml/badge.svg)](https://github.com/zvndev/powdb/actions/workflows/bench.yml) [![crates.io](https://img.shields.io/crates/v/powdb-cli.svg)](https://crates.io/crates/powdb-cli) [![docs.rs](https://img.shields.io/docsrs/powdb-query)](https://docs.rs/powdb-query) [![MSRV](https://img.shields.io/badge/MSRV-1.93-blue)](https://github.com/zvndev/powdb/blob/main/Cargo.toml) @@ -30,7 +29,7 @@ Evaluating PowDB? Start with the honest comparison: [PowDB vs SQLite -- when to | Filter + project | `SELECT name, age FROM User WHERE age > 25` | `User filter .age > 25 { .name, .age }` | | Sort + limit | `SELECT * FROM User ORDER BY age DESC LIMIT 10` | `User order .age desc limit 10` | | Aggregate with filter | `SELECT AVG(age) FROM User WHERE city = 'NYC'` | `avg(User filter .city = "NYC" { .age })` | -| Group + having | `SELECT status, COUNT(*) FROM User GROUP BY status HAVING COUNT(*) > 5` | `User group .status having count(*) > 5 { .status, count(*) }` | +| Group + having | `SELECT status, COUNT(name) FROM User GROUP BY status HAVING COUNT(name) > 5` | `User group .status having count(.name) > 5 { .status, n: count(.name) }` | PowQL uses `.field` dot syntax for column references, `:=` for assignments, and `"double quotes"` for strings. The pipeline reads like a sentence: *"User, filter age greater than 25, order by name, limit 10, give me name and age."* @@ -43,7 +42,7 @@ Full language reference: [docs/POWQL.md](https://github.com/zvndev/powdb/blob/ma cargo install powdb-cli cargo install powdb-server -# TypeScript client (Node 18+) +# TypeScript client (Node 18+) — versions independently of the server crates (currently 0.3.x) npm install @zvndev/powdb-client # Prebuilt binaries (linux x86_64, macos aarch64) @@ -201,10 +200,14 @@ if (result.kind === "rows") console.table(result.rows); | Variable | Default | Description | |---|---|---| | `POWDB_PORT` | `5433` | TCP port for the server | +| `POWDB_BIND` | `127.0.0.1` | Interface to bind; set `0.0.0.0` behind a platform proxy (Fly, Railway) | | `POWDB_DATA` | `./powdb_data` | Data directory (heap files, WAL, catalog, indexes) | | `POWDB_PASSWORD` | *(none)* | Shared password required on connect when no named users are defined (set as env var) | | `POWDB_ADMIN_USER` / `POWDB_ADMIN_PASSWORD` | *(none)* | Bootstrap an `admin` user on startup when both are set and that user does not yet exist (password never logged) | +| `POWDB_TLS_CERT` / `POWDB_TLS_KEY` | *(none)* | Paths to PEM cert + key; when both are set the server serves TLS | | `POWDB_REQUIRE_TLS` | *(off)* | When set (`1`/`true`), refuse to start if a password is configured without TLS | +| `POWDB_IDLE_TIMEOUT` | `300` | Seconds before an idle connection is closed | +| `POWDB_QUERY_TIMEOUT` | `30` | Per-query timeout in seconds | | `POWDB_QUERY_MEMORY_LIMIT` | `268435456` | Per-query memory budget in bytes (256 MiB); over-budget queries error instead of OOM-killing the server | | `RUST_LOG` | `info` | Log level (`debug`, `trace` for per-query timings) | @@ -232,6 +235,7 @@ For a self-hostable starting point, see [`examples/deploy/fly.toml`](https://git - Memory-mapped reads (zero-syscall scan path) - Compiled integer predicates (branch-free filter at the byte level) - Thread-safe concurrent reads via pread(2)/pwrite(2) +- Backup & restore: full + incremental + coarse point-in-time recovery (offline; `powdb-cli backup` / `restore` — see [docs/backup-and-restore.md](docs/backup-and-restore.md)) **Query engine** - PowQL parser + planner + executor with plan cache (FNV-1a hashing, literal substitution) @@ -273,6 +277,8 @@ For a self-hostable starting point, see [`examples/deploy/fly.toml`](https://git crates/ storage/ Heap files, B+tree, WAL, catalog, page cache, row encoding query/ Lexer, parser, planner, executor (Engine), plan cache + auth/ User store, roles, argon2id password hashing + backup/ Offline backup/restore (full, incremental, PITR) server/ Tokio TCP server + binary wire protocol cli/ Interactive REPL (embedded + remote modes) bench/ Criterion benchmarks + regression gate @@ -283,13 +289,15 @@ The engine is `powdb_query::executor::Engine`. It owns a `Catalog` (which owns ` ## Benchmarks -PowDB has a CI-enforced regression gate that blocks PRs to `main` if any workload regresses beyond its threshold. Run locally: +PowDB has a benchmark regression gate that compares every workload against checked-in baselines. Run it locally before and after touching a hot path: ```bash cargo bench -p powdb-bench # criterion suite (~60s) cargo run --release -p powdb-bench --bin compare # regression gate ``` +The gate also runs on-demand in CI via `workflow_dispatch` (`.github/workflows/bench.yml`) — it is **not** a required PR gate, because shared-runner noise makes it unreliable as a blocking check. The required PR gates live in `ci.yml`. + Run the PowDB vs SQLite comparison bench: ```bash diff --git a/RELEASES.md b/RELEASES.md index 2219872..1ffff78 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -3,9 +3,10 @@ Every PowDB release ships to the following registries and platforms. When cutting a release, follow the checklist at the bottom. -> **Current release: v0.4.4** (all four crates live on crates.io). +> **Current release: v0.4.5** (all six crates live on crates.io, including the +> new `powdb-auth` and `powdb-backup`). > **v0.4.1, v0.4.2, and v0.4.3 are yanked** for crash-recovery data-loss bugs; -> 0.4.4 fixes them and adds a standing durability regression suite. See +> 0.4.4 fixed them and added a standing durability regression suite. See > `CHANGELOG.md`. ## Registries @@ -13,10 +14,13 @@ When cutting a release, follow the checklist at the bottom. | Target | Package | Registry URL | |--------|---------|-------------| | **crates.io** | `powdb-storage` | https://crates.io/crates/powdb-storage | +| **crates.io** | `powdb-auth` | https://crates.io/crates/powdb-auth | | **crates.io** | `powdb-query` | https://crates.io/crates/powdb-query | +| **crates.io** | `powdb-backup` | https://crates.io/crates/powdb-backup | | **crates.io** | `powdb-server` | https://crates.io/crates/powdb-server | | **crates.io** | `powdb-cli` | https://crates.io/crates/powdb-cli | | **npm** | `@zvndev/powdb-client` | https://www.npmjs.com/package/@zvndev/powdb-client | +| **ghcr.io** | `ghcr.io/zvn-dev/powdb` (Docker image, `latest` + `vX.Y.Z` tags) | https://github.com/orgs/ZVN-DEV/packages | ## GitHub Releases @@ -35,9 +39,11 @@ when a `v*` tag is pushed. Inter-crate dependencies require publishing in this order: 1. `powdb-storage` (no inter-crate deps) -2. `powdb-query` (depends on storage) -3. `powdb-server` (depends on storage + query) -4. `powdb-cli` (depends on storage + query + server) +2. `powdb-auth` (no inter-crate deps) +3. `powdb-query` (depends on storage) +4. `powdb-backup` (depends on storage + query) +5. `powdb-server` (depends on storage + query + auth) +6. `powdb-cli` (depends on storage + query + server + backup + auth) Non-publishable crates (`publish = false`): `powdb-compare`, `powdb-bench`, `powdb-query-fuzz`. @@ -45,12 +51,14 @@ Non-publishable crates (`publish = false`): `powdb-compare`, `powdb-bench`, `pow ``` [ ] Update workspace version in root Cargo.toml -[ ] Update inter-crate dep versions in query/server/cli Cargo.toml +[ ] Update inter-crate dep versions in query/backup/server/cli Cargo.toml [ ] Update clients/ts/package.json version [ ] Update CHANGELOG.md [ ] Commit: "chore: release vX.Y.Z" [ ] cargo publish -p powdb-storage +[ ] cargo publish -p powdb-auth [ ] cargo publish -p powdb-query +[ ] cargo publish -p powdb-backup [ ] cargo publish -p powdb-server [ ] cargo publish -p powdb-cli [ ] cd clients/ts && npm publish --access public diff --git a/SECURITY.md b/SECURITY.md index 33c8103..c1d757d 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -4,7 +4,8 @@ | Version | Supported | | --------------- | ------------------ | -| 0.4.4 | :white_check_mark: | +| 0.4.5 | :white_check_mark: | +| 0.4.4 | :x: (superseded) | | 0.4.1 – 0.4.3 | :x: (yanked) | | 0.4.0 | :white_check_mark: | | 0.3.x | :white_check_mark: | @@ -14,7 +15,7 @@ > **v0.4.1, v0.4.2, and v0.4.3 are yanked** for data-loss bugs in crash > recovery and have been replaced by **v0.4.4**, which adds a permanent > durability regression suite. If you are on any of those three versions, -> upgrade to 0.4.4. See `CHANGELOG.md` for details. +> upgrade to the latest release (0.4.5). See `CHANGELOG.md` for details. ## Reporting a Vulnerability @@ -52,15 +53,20 @@ When both are set, the server requires TLS for all connections. When unset, the ## Authentication -PowDB uses single-password authentication via the `POWDB_PASSWORD` environment variable. When set, clients must authenticate before executing queries. +PowDB supports two authentication modes: + +1. **Shared password** — set the `POWDB_PASSWORD` environment variable. All clients authenticate with the same shared secret. Applies only when no named users are defined. +2. **Named users with roles** (since 0.4.5) — users with `admin`, `readwrite`, or `readonly` roles, managed via `powdb-cli useradd` / `passwd` / `userdel`. Passwords are stored as argon2id hashes only (`auth.json` in the data directory, `0600` on Unix). When `POWDB_ADMIN_USER` and `POWDB_ADMIN_PASSWORD` are both set, the server bootstraps an initial admin on startup without the CLI. Once any user is defined, the shared password is no longer used. + +In both modes: - **Rate limiting**: authentication attempts are rate-limited to prevent brute-force attacks. - **Pre-auth payload limits**: the server enforces frame size limits on unauthenticated connections to prevent resource exhaustion. - **Connection limits**: the server enforces a maximum number of concurrent connections. -There is no per-user auth or RBAC. The password should be treated as a shared secret for all clients connecting to a given server instance. +> **Note on the `readonly` role:** in releases up to and including 0.4.5, role storage is in place but read-only restrictions are **not enforced** at the query layer — do not rely on the `readonly` role as a security boundary against writes on those versions. Enforcement at the server dispatch layer ships in the next release: write statements from `readonly` users are rejected with `permission denied`, and unknown roles fail closed. ## Known Limitations -- Authentication is single-password. There is no per-user auth or RBAC. -- The query parser has a nesting depth limit but no query timeout mechanism yet. +- Roles are coarse (`admin` / `readwrite` / `readonly`). There are no per-table ACLs, row-level security, or multi-tenant isolation; `readonly` enforcement is absent in ≤0.4.5 (see note above). +- The query parser has a nesting depth limit. Runaway queries are bounded by `POWDB_QUERY_TIMEOUT` (default 30s) and the per-query memory budget (`POWDB_QUERY_MEMORY_LIMIT`). diff --git a/clients/ts/CHANGELOG.md b/clients/ts/CHANGELOG.md index 352e4bc..a74bc18 100644 --- a/clients/ts/CHANGELOG.md +++ b/clients/ts/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to `@zvndev/powdb-client`. | Client version | Compatible PowDB server | Notes | |---|---|---| -| 0.3.x | 0.3.x – 0.4.x | Wire protocol v1. The client warns only on a major-version mismatch, so any `0.x` server connects. Minor server bumps may add new opcodes; the client tolerates unknown response codes by surfacing `PowDBError`. Pin both ends. | +| 0.3.x | 0.3.x – 0.4.x | Wire protocol v1. The client warns only on a major-version mismatch, so any `0.x` server connects. Minor server bumps may add new opcodes; the client tolerates unknown response codes by surfacing `PowDBError`. Pin both ends. **Caveat:** the 0.3.x client has no `user` option, so it cannot authenticate to a 0.4.5+ server running in **multi-user mode** (the server requires a username once any named user is defined). Shared-password mode works fine. | The client warns on major-version mismatch with the server during the handshake. Within `0.x`, treat any minor-version skew between client and diff --git a/clients/ts/README.md b/clients/ts/README.md index 367de10..0a49576 100644 --- a/clients/ts/README.md +++ b/clients/ts/README.md @@ -243,6 +243,11 @@ Returns a `Promise`. Options: | `connectTimeoutMs` | `number` | `5000` | Connection timeout in milliseconds | | `tls` | `boolean \| tls.ConnectionOptions` | `false` | Enable TLS; `true` uses system defaults, or pass a `tls.connect` options object | +> **Multi-user servers:** PowDB server 0.4.5 added named users with roles. The +> 0.3.x client has no `user` option, so it **cannot authenticate to a server +> running in multi-user mode** (the server requires a username once any user +> is defined). Shared-password mode (`POWDB_PASSWORD`) works as before. + ### `client.query(query, opts?)` Sends a PowQL query and returns a `Promise`: diff --git a/clients/ts/demo/demo.ts b/clients/ts/demo/demo.ts index 52a3aad..eeaaaa6 100644 --- a/clients/ts/demo/demo.ts +++ b/clients/ts/demo/demo.ts @@ -28,13 +28,14 @@ async function main() { try { // Use a unique table name per run so reruns don't collide on a - // persistent server. PowQL doesn't have DROP TABLE (yet). + // persistent server. (PowQL has `drop `, but a unique name keeps + // the demo side-effect free even if a run is interrupted.) const tableName = `Demo${Date.now().toString(36)}`; const table = ident(tableName); console.log(`→ creating type ${tableName}`); await run( client, - powql`type ${table} { required name: string, required age: int, city: string }`, + powql`type ${table} { required name: str, required age: int, city: str }`, ); console.log(`→ inserting rows`); @@ -101,6 +102,9 @@ function printResult(result: QueryResult): void { case "ok": console.log(` → ok (${result.affected} affected)`); break; + case "message": + console.log(` → ${result.message}`); + break; } } diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index cf7c6a7..f99072b 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -1205,6 +1205,19 @@ fn print_local_result(result: &QueryResult) { } } +/// Render one wire cell for display. The server serializes NULL as the +/// bareword "null" (the sentinel the TS client's typed decoder matches); +/// remote mode renders it as `NULL`, matching the embedded REPL. A string +/// column whose *value* is literally "null" is indistinguishable on the +/// untyped wire — same tradeoff the TS client documents. +fn render_remote_cell(cell: &str) -> String { + if cell == "null" { + "NULL".into() + } else { + cell.into() + } +} + fn print_remote_result(msg: &Message) { match msg { Message::ResultRows { columns, rows } => { @@ -1212,10 +1225,14 @@ fn print_remote_result(msg: &Message) { println!("(empty set)"); return; } - print_table(columns, rows); + let rendered: Vec> = rows + .iter() + .map(|row| row.iter().map(|c| render_remote_cell(c)).collect()) + .collect(); + print_table(columns, &rendered); } Message::ResultScalar { value } => { - println!("{value}"); + println!("{}", render_remote_cell(value)); } Message::ResultOk { affected } => { println!( @@ -1286,3 +1303,20 @@ fn format_value(v: &Value) -> String { Value::Empty => "NULL".into(), } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn remote_null_sentinel_renders_as_null_like_embedded() { + // The wire sends NULL as the bareword "null"; remote display must + // match the embedded REPL's `NULL`. + assert_eq!(render_remote_cell("null"), "NULL"); + assert_eq!(format_value(&Value::Empty), "NULL"); + // Ordinary values pass through untouched. + assert_eq!(render_remote_cell("42"), "42"); + assert_eq!(render_remote_cell(""), ""); + assert_eq!(render_remote_cell("NULL"), "NULL"); + } +} diff --git a/crates/query/src/executor/plan_exec.rs b/crates/query/src/executor/plan_exec.rs index 8aa1076..c0ebea6 100644 --- a/crates/query/src/executor/plan_exec.rs +++ b/crates/query/src/executor/plan_exec.rs @@ -2914,6 +2914,26 @@ pub(super) fn execute_window( std::cmp::Ordering::Equal }); + // SQL window-frame semantics: with no `order` clause the frame for an + // aggregate window is the ENTIRE partition, not the running prefix. + // The loop below computes running values; for the no-order case we + // back-fill every row of a partition with the partition's final + // (i.e. complete) aggregate once its boundary is reached. Ranking + // functions are untouched — row_number/rank/dense_rank are inherently + // positional. + let whole_partition_frame = wdef.order_by.is_empty() + && matches!( + wdef.function, + WindowFunc::Sum + | WindowFunc::Avg + | WindowFunc::Count + | WindowFunc::Min + | WindowFunc::Max + ); + // Original row indices of the partition currently being scanned + // (only tracked when back-filling is needed). + let mut partition_row_indices: Vec = Vec::new(); + // Compute window values in sorted order, tracking partition boundaries. let mut win_values: Vec = vec![Value::Empty; n]; let mut partition_start = 0usize; @@ -2943,6 +2963,15 @@ pub(super) fn execute_window( }; if new_partition { + // No-order aggregate frame: the partition that just ended is + // complete, so its final running value IS the whole-partition + // aggregate. Back-fill it onto every row of that partition. + if whole_partition_frame && sorted_pos > 0 { + let final_v = win_values[indices[sorted_pos - 1]].clone(); + for ri in partition_row_indices.drain(..) { + win_values[ri] = final_v.clone(); + } + } partition_start = sorted_pos; running_count = 0; running_int_sum = 0; @@ -3071,6 +3100,17 @@ pub(super) fn execute_window( prev_order_key = Some(current_order_key); win_values[row_idx] = value; + if whole_partition_frame { + partition_row_indices.push(row_idx); + } + } + + // Back-fill the final partition (the loop only flushes at boundaries). + if whole_partition_frame && n > 0 { + let final_v = win_values[indices[n - 1]].clone(); + for ri in partition_row_indices.drain(..) { + win_values[ri] = final_v.clone(); + } } // Append the computed window column to each row. diff --git a/crates/query/src/executor/tests.rs b/crates/query/src/executor/tests.rs index f5e8abc..bfc1509 100644 --- a/crates/query/src/executor/tests.rs +++ b/crates/query/src/executor/tests.rs @@ -4172,3 +4172,131 @@ fn test_memory_limit_default_allows_normal_query() { _ => panic!("expected rows"), } } + +// --------------------------------------------------------------------------- +// Window aggregates without `order`: frame must be the ENTIRE partition, +// not a running prefix. (`avg(.sal) over (partition .dept)` used to return +// 10/15/20 for salaries 10/20/30 instead of 20/20/20.) +// --------------------------------------------------------------------------- + +fn window_engine() -> Engine { + let id = TEST_COUNTER.fetch_add(1, Ordering::SeqCst); + let dir = std::env::temp_dir().join(format!("powdb_win_{}_{}", std::process::id(), id)); + let mut engine = Engine::new(&dir).unwrap(); + engine + .execute_powql("type Emp { required name: str, required dept: str, required sal: int }") + .unwrap(); + for (name, dept, sal) in [ + ("a", "eng", 10), + ("b", "eng", 20), + ("c", "eng", 30), + ("d", "ops", 100), + ("e", "ops", 300), + ] { + engine + .execute_powql(&format!( + r#"insert Emp {{ name := "{name}", dept := "{dept}", sal := {sal} }}"# + )) + .unwrap(); + } + engine +} + +/// Extract (name → window value) pairs from a two-column result. +fn window_col_by_name(result: QueryResult) -> std::collections::HashMap { + match result { + QueryResult::Rows { columns, rows } => { + let name_idx = columns.iter().position(|c| c == "name").unwrap(); + let win_idx = columns.len() - 1; + rows.into_iter() + .map(|r| { + let name = match &r[name_idx] { + Value::Str(s) => s.clone(), + other => panic!("expected name string, got {other:?}"), + }; + (name, r[win_idx].clone()) + }) + .collect() + } + other => panic!("expected rows, got {other:?}"), + } +} + +#[test] +fn test_window_agg_without_order_uses_whole_partition() { + let mut engine = window_engine(); + let result = engine + .execute_powql("Emp { .name, davg: avg(.sal) over (partition .dept) }") + .unwrap(); + let by_name = window_col_by_name(result); + // eng partition avg = (10+20+30)/3 = 20 for EVERY row. + for n in ["a", "b", "c"] { + assert_eq!(by_name[n], Value::Float(20.0), "row {n}"); + } + // ops partition avg = (100+300)/2 = 200 for both rows. + for n in ["d", "e"] { + assert_eq!(by_name[n], Value::Float(200.0), "row {n}"); + } +} + +#[test] +fn test_window_sum_count_min_max_without_order_whole_partition() { + let mut engine = window_engine(); + + let by_name = window_col_by_name( + engine + .execute_powql("Emp { .name, dsum: sum(.sal) over (partition .dept) }") + .unwrap(), + ); + for n in ["a", "b", "c"] { + assert_eq!(by_name[n], Value::Int(60), "sum row {n}"); + } + for n in ["d", "e"] { + assert_eq!(by_name[n], Value::Int(400), "sum row {n}"); + } + + let by_name = window_col_by_name( + engine + .execute_powql("Emp { .name, dcnt: count(.sal) over (partition .dept) }") + .unwrap(), + ); + for n in ["a", "b", "c"] { + assert_eq!(by_name[n], Value::Int(3), "count row {n}"); + } + + let by_name = window_col_by_name( + engine + .execute_powql("Emp { .name, dmin: min(.sal) over (partition .dept) }") + .unwrap(), + ); + for n in ["a", "b", "c"] { + assert_eq!(by_name[n], Value::Int(10), "min row {n}"); + } + + let by_name = window_col_by_name( + engine + .execute_powql("Emp { .name, dmax: max(.sal) over (partition .dept) }") + .unwrap(), + ); + for n in ["a", "b", "c"] { + assert_eq!(by_name[n], Value::Int(30), "max row {n}"); + } + for n in ["d", "e"] { + assert_eq!(by_name[n], Value::Int(300), "max row {n}"); + } +} + +#[test] +fn test_window_agg_with_order_keeps_running_frame() { + // WITH an explicit `order`, the running (rows-so-far) frame is the + // existing documented behavior — it must not change. + let mut engine = window_engine(); + let by_name = window_col_by_name( + engine + .execute_powql("Emp { .name, ravg: avg(.sal) over (partition .dept order .sal) }") + .unwrap(), + ); + assert_eq!(by_name["a"], Value::Float(10.0)); + assert_eq!(by_name["b"], Value::Float(15.0)); + assert_eq!(by_name["c"], Value::Float(20.0)); +} diff --git a/crates/query/tests/oversized_rows.rs b/crates/query/tests/oversized_rows.rs new file mode 100644 index 0000000..fe7064d --- /dev/null +++ b/crates/query/tests/oversized_rows.rs @@ -0,0 +1,135 @@ +//! Oversized-row regression tests (remote-DoS fix). +//! +//! With `panic = "abort"` in the release profile, a panicking insert path +//! kills the whole server process. An encoded row larger than one 4KB +//! slotted page used to hit `expect("row too large for empty page")` in +//! `HeapFile::insert`. These tests pin the fixed behavior: oversized rows +//! are a clean query error and the engine stays fully usable. + +use powdb_query::executor::Engine; +use powdb_query::result::QueryResult; + +fn temp_engine(name: &str) -> (Engine, std::path::PathBuf) { + let dir = std::env::temp_dir().join(format!( + "powdb_oversized_{name}_{}_{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + std::fs::create_dir_all(&dir).unwrap(); + let engine = Engine::new(&dir).unwrap(); + (engine, dir) +} + +fn big_string(n: usize) -> String { + "x".repeat(n) +} + +#[test] +fn oversized_insert_is_clean_error_and_engine_survives() { + let (mut engine, dir) = temp_engine("insert"); + engine + .execute_powql("type T { required id: int, b: str }") + .unwrap(); + + // 5000-byte string > 4KB page capacity → must be a clean error. + let q = format!(r#"insert T {{ id := 1, b := "{}" }}"#, big_string(5000)); + let err = engine + .execute_powql(&q) + .expect_err("oversized insert must fail"); + assert!( + err.to_string().contains("row too large"), + "unexpected error: {err}" + ); + + // Engine keeps working: normal insert + query succeed. + engine + .execute_powql(r#"insert T { id := 2, b := "small" }"#) + .unwrap(); + let res = engine.execute_powql("count(T)").unwrap(); + match res { + QueryResult::Scalar(v) => assert_eq!(format!("{v:?}"), "Int(1)"), + other => panic!("expected scalar count, got {other:?}"), + } + drop(engine); + std::fs::remove_dir_all(&dir).ok(); +} + +#[test] +fn oversized_update_is_clean_error_and_row_intact() { + let (mut engine, dir) = temp_engine("update"); + engine + .execute_powql("type T { required id: int, b: str }") + .unwrap(); + engine + .execute_powql(r#"insert T { id := 1, b := "original" }"#) + .unwrap(); + + // Oversized new value on update must fail cleanly... + let q = format!( + r#"T filter .id = 1 update {{ b := "{}" }}"#, + big_string(5000) + ); + let err = engine + .execute_powql(&q) + .expect_err("oversized update must fail"); + assert!( + err.to_string().contains("row too large"), + "unexpected error: {err}" + ); + + // ...and the original row must be untouched. + let res = engine.execute_powql("T filter .id = 1").unwrap(); + match res { + QueryResult::Rows { rows, .. } => { + assert_eq!(rows.len(), 1, "row must still exist"); + let joined = format!("{:?}", rows[0]); + assert!( + joined.contains("original"), + "row value must be unchanged: {joined}" + ); + } + other => panic!("expected rows, got {other:?}"), + } + drop(engine); + std::fs::remove_dir_all(&dir).ok(); +} + +#[test] +fn oversized_update_does_not_poison_wal_replay() { + // A failed oversized update must not leave a WAL record that breaks (or + // mutates state during) recovery on the next open. + let (mut engine, dir) = temp_engine("replay"); + engine + .execute_powql("type T { required id: int, b: str }") + .unwrap(); + engine + .execute_powql(r#"insert T { id := 1, b := "original" }"#) + .unwrap(); + let q = format!( + r#"T filter .id = 1 update {{ b := "{}" }}"#, + big_string(5000) + ); + engine + .execute_powql(&q) + .expect_err("oversized update must fail"); + // Run one more statement so any buffered WAL bytes get group-committed. + engine.execute_powql("count(T)").unwrap(); + drop(engine); + + // Reopen: replay must succeed and show the original value. + let mut engine = Engine::new(&dir).expect("reopen after failed oversized update"); + let res = engine.execute_powql("T filter .id = 1").unwrap(); + match res { + QueryResult::Rows { rows, .. } => { + assert_eq!(rows.len(), 1); + let joined = format!("{:?}", rows[0]); + assert!(joined.contains("original"), "row corrupted: {joined}"); + } + other => panic!("expected rows, got {other:?}"), + } + drop(engine); + std::fs::remove_dir_all(&dir).ok(); +} diff --git a/crates/server/src/handler.rs b/crates/server/src/handler.rs index b9234c1..46ccd8d 100644 --- a/crates/server/src/handler.rs +++ b/crates/server/src/handler.rs @@ -1,5 +1,5 @@ use crate::protocol::Message; -use powdb_auth::UserStore; +use powdb_auth::{Permission, Role, UserStore}; use powdb_query::executor::{is_read_only_statement, Engine}; use powdb_query::parser; use powdb_query::result::{QueryError, QueryResult}; @@ -81,15 +81,48 @@ fn constant_time_eq(a: &[u8], b: &[u8]) -> bool { diff == 0 } -/// An authenticated connection's identity, carried alongside the session for -/// later per-operation RBAC enforcement (Slice 3). For now it is bound at -/// connect time and logged; it is not yet consulted per query. +/// An authenticated connection's identity. Bound at connect time and consulted +/// on every query by [`dispatch_query`] to enforce the user's role: a +/// `readonly` principal may only execute read statements. #[derive(Debug, Clone, PartialEq, Eq)] pub struct Principal { pub name: String, pub role: String, } +/// Whether `role` grants the `Write` permission. Unknown role names fail +/// closed (no write). Shared-password / open / embedded modes never construct +/// a [`Principal`], so they are unaffected by this gate. +fn role_can_write(role: &str) -> bool { + Role::builtin(role).is_some_and(|r| r.allows(Permission::Write)) +} + +/// Enforce the principal's role against a parsed statement. Returns an error +/// for any non-read statement (insert/update/delete/upsert/DDL/view ops/ +/// transaction control) when the role does not grant `Write`. +/// +/// Classification uses the parsed AST via +/// [`powdb_query::executor::is_read_only_statement`] — the exact same +/// classifier the RwLock read/write split relies on — so the permission +/// boundary and the concurrency boundary can never disagree. +fn check_statement_permitted( + principal: Option<&Principal>, + stmt: &powdb_query::ast::Statement, +) -> Result<(), QueryError> { + let Some(p) = principal else { + // No per-user identity (shared-password or open mode): full access, + // byte-identical to the pre-RBAC behavior. + return Ok(()); + }; + if is_read_only_statement(stmt) || role_can_write(&p.role) { + return Ok(()); + } + Err(QueryError::Execution(format!( + "permission denied: role '{}' cannot execute write statements", + p.role + ))) +} + /// Result of the connect-time authentication decision. #[derive(Debug, Clone, PartialEq, Eq)] pub enum AuthOutcome { @@ -169,6 +202,8 @@ const SAFE_ERROR_PREFIXES: &[&str] = &[ "cannot", "no such", "already exists", + "permission denied", + "row too large", ]; /// Sanitize an error message before sending it to the client. @@ -219,9 +254,25 @@ pub struct ConnOpts<'a> { /// Execute a query against the engine under the RwLock. Read-only /// statements acquire `.read()` so concurrent SELECTs can scan in /// parallel; mutations acquire `.write()`. -fn dispatch_query(engine: &Arc>, query: &str) -> Result { +/// +/// When `principal` is `Some`, the user's role is enforced first: a role +/// without the `Write` permission (i.e. `readonly`) gets a clean +/// "permission denied" error for any non-read statement, before any lock +/// is taken or any engine state is touched. +fn dispatch_query( + engine: &Arc>, + query: &str, + principal: Option<&Principal>, +) -> Result { let stmt_result = parser::parse(query).map_err(|e| e.to_string()); + // Role enforcement happens on the parsed AST. Statements that fail to + // parse fall through — the engine returns the parse error itself and + // can never execute anything for them. + if let Ok(stmt) = &stmt_result { + check_statement_permitted(principal, stmt)?; + } + let can_try_read = matches!(&stmt_result, Ok(s) if is_read_only_statement(s)); if can_try_read { let res = { @@ -298,9 +349,9 @@ where } }; - // The authenticated identity for this connection. Bound at connect time and - // carried for later per-operation RBAC enforcement (Slice 3). - let _principal: Option; + // The authenticated identity for this connection. Bound at connect time + // and enforced on every query by `dispatch_query`. + let principal: Option; match connect_msg { Message::Connect { db_name, @@ -339,12 +390,14 @@ where write_msg(&mut writer, &err).await; return; } - AuthOutcome::Authenticated { principal } => { + AuthOutcome::Authenticated { + principal: auth_principal, + } => { // Auth succeeded — clear any prior failure count. if let (Some(limiter), Some(ip)) = (rate_limiter, peer_ip) { clear_auth_failures(limiter, ip); } - match &principal { + match &auth_principal { Some(p) => { info!(peer = %peer, db = %db_name, user = %p.name, role = %p.role, "authenticated"); } @@ -352,7 +405,7 @@ where info!(peer = %peer, db = %db_name, "client connected"); } } - _principal = principal; + principal = auth_principal; } } @@ -424,7 +477,8 @@ where let handle = tokio::task::spawn_blocking({ let engine = engine.clone(); let query = query.clone(); - move || dispatch_query(&engine, &query) + let principal = principal.clone(); + move || dispatch_query(&engine, &query, principal.as_ref()) }); let abort_handle = handle.abort_handle(); match tokio::time::timeout(query_timeout, handle).await { @@ -496,7 +550,12 @@ fn value_to_display(v: &Value) -> String { u[0], u[1], u[2], u[3], u[4], u[5], u[6], u[7], u[8], u[9], u[10], u[11], u[12], u[13], u[14], u[15]), Value::Bytes(b) => format!("<{} bytes>", b.len()), - Value::Empty => "{}".into(), + // NULL is serialized as the bareword "null" on the wire. This is the + // sentinel the TypeScript client's typed-row decoder already + // documents and matches (`coerceValue` treats the exact token + // "null" as NULL for non-str columns); the previous "{}" rendering + // was a bug that neither the TS client nor the CLI recognized. + Value::Empty => "null".into(), } } @@ -504,6 +563,85 @@ fn value_to_display(v: &Value) -> String { mod tests { use super::*; + // ---- Wire NULL rendering (Fix: remote protocol rendered NULL as `{}`) ---- + + #[test] + fn null_serializes_as_null_bareword_on_wire() { + assert_eq!(value_to_display(&Value::Empty), "null"); + } + + // ---- Role enforcement (Fix: readonly role was not enforced) ---- + + fn parsed(q: &str) -> powdb_query::ast::Statement { + parser::parse(q).unwrap() + } + + fn principal(role: &str) -> Option { + Some(Principal { + name: "u".into(), + role: role.into(), + }) + } + + #[test] + fn readonly_can_read_but_not_write() { + let p = principal("readonly"); + // Reads pass. + assert!(check_statement_permitted(p.as_ref(), &parsed("User")).is_ok()); + assert!(check_statement_permitted(p.as_ref(), &parsed("count(User)")).is_ok()); + assert!(check_statement_permitted(p.as_ref(), &parsed("explain User")).is_ok()); + // Writes, DDL, and transaction control are denied. + for q in [ + r#"insert User { name := "x" }"#, + "User filter .id = 1 update { age := 2 }", + "User filter .id = 1 delete", + "drop User", + "alter User add column c: str", + "type T { required id: int }", + "begin", + "commit", + "rollback", + ] { + let err = check_statement_permitted(p.as_ref(), &parsed(q)) + .expect_err(&format!("must deny: {q}")); + assert!( + err.to_string().contains("permission denied"), + "unexpected error for {q}: {err}" + ); + } + } + + #[test] + fn readwrite_and_admin_have_full_query_access() { + for role in ["readwrite", "admin"] { + let p = principal(role); + assert!(check_statement_permitted(p.as_ref(), &parsed("User")).is_ok()); + assert!(check_statement_permitted( + p.as_ref(), + &parsed(r#"insert User { name := "x" }"#) + ) + .is_ok()); + assert!(check_statement_permitted(p.as_ref(), &parsed("drop User")).is_ok()); + } + } + + #[test] + fn unknown_role_fails_closed_for_writes() { + let p = principal("mystery"); + assert!(check_statement_permitted(p.as_ref(), &parsed("User")).is_ok()); + assert!( + check_statement_permitted(p.as_ref(), &parsed(r#"insert User { name := "x" }"#)) + .is_err() + ); + } + + #[test] + fn no_principal_means_full_access() { + // Shared-password / open mode: no per-user identity, no restriction. + assert!(check_statement_permitted(None, &parsed("drop User")).is_ok()); + assert!(check_statement_permitted(None, &parsed(r#"insert User { name := "x" }"#)).is_ok()); + } + fn store_with_alice() -> UserStore { let mut s = UserStore::new(); s.create_user("alice", "pw", "readwrite").unwrap(); diff --git a/crates/server/tests/integration.rs b/crates/server/tests/integration.rs index c0bf301..c7257a8 100644 --- a/crates/server/tests/integration.rs +++ b/crates/server/tests/integration.rs @@ -335,6 +335,25 @@ async fn test_empty_store_shared_password_fallback() { stream.write_all(&frame).await.unwrap(); let resp = read_response(&mut stream).await; assert_eq!(resp[0], 0x02, "correct shared password should connect"); + + // Shared-password mode has no per-user role: writes must keep + // working exactly as before role enforcement existed. + stream + .write_all(&encode_query("type Item { required name: str }")) + .await + .unwrap(); + let resp = read_response(&mut stream).await; + assert!( + resp[0] == 0x09 || resp[0] == 0x0B, + "shared-password DDL must succeed, got 0x{:02X}", + resp[0] + ); + stream + .write_all(&encode_query(r#"insert Item { name := "widget" }"#)) + .await + .unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!(resp[0], 0x09, "shared-password insert must succeed"); } // Wrong shared password → ERROR. @@ -354,3 +373,181 @@ async fn test_empty_store_shared_password_fallback() { handle.abort(); std::fs::remove_dir_all(&data_dir).ok(); } + +/// Fix 2 (authorization bypass): the `readonly` role must be enforced at the +/// server dispatch boundary. A readonly user can run read statements but every +/// write (insert/update/delete/DDL/transaction control) is rejected with a +/// clean "permission denied" error — and the connection stays alive. +/// `readwrite` and `admin` users keep full query access. +#[tokio::test] +async fn test_readonly_role_enforced_over_tcp() { + use powdb_auth::UserStore; + use powdb_server::protocol::Message; + use std::sync::{Arc, RwLock}; + + let test_id = std::process::id(); + let port = 18500 + (test_id % 1000) as u16; + let data_dir = std::env::temp_dir().join(format!("powdb_rbac_{test_id}")); + std::fs::create_dir_all(&data_dir).unwrap(); + + let mut store = UserStore::new(); + store.create_user("root", "pw", "admin").unwrap(); + store.create_user("rw", "pw", "readwrite").unwrap(); + store.create_user("ro", "pw", "readonly").unwrap(); + let users = Arc::new(store); + + let data_dir_str = data_dir.to_str().unwrap().to_string(); + let addr = format!("127.0.0.1:{port}"); + let bind_addr = addr.clone(); + + let handle = tokio::spawn(async move { + let engine = + powdb_query::executor::Engine::new(std::path::Path::new(&data_dir_str)).unwrap(); + let engine = Arc::new(RwLock::new(engine)); + let listener = tokio::net::TcpListener::bind(&bind_addr).await.unwrap(); + loop { + let (stream, peer) = listener.accept().await.unwrap(); + let eng = engine.clone(); + let users = users.clone(); + let (_, mut rx) = tokio::sync::watch::channel(false); + tokio::spawn(async move { + powdb_server::handler::handle_connection( + stream, + powdb_server::handler::ConnOpts { + engine: eng, + expected_password: None, + users, + shutdown_rx: &mut rx, + idle_timeout: Duration::from_secs(300), + query_timeout: Duration::from_secs(30), + rate_limiter: None, + peer_addr: Some(peer), + }, + ) + .await; + }); + } + }); + + tokio::time::sleep(Duration::from_millis(100)).await; + + // Admin seeds the schema + one row. + { + let mut stream = TcpStream::connect(&addr).await.unwrap(); + stream + .write_all(&encode_connect_user("testdb", "pw", "root")) + .await + .unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!(resp[0], 0x02, "admin should connect"); + + stream + .write_all(&encode_query("type User { required name: str, age: int }")) + .await + .unwrap(); + let resp = read_response(&mut stream).await; + assert!( + resp[0] == 0x09 || resp[0] == 0x0B, + "admin DDL should succeed, got 0x{:02X}", + resp[0] + ); + + stream + .write_all(&encode_query( + r#"insert User { name := "Alice", age := 30 }"#, + )) + .await + .unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!(resp[0], 0x09, "admin insert should succeed"); + } + + // Readonly user: reads OK, every write shape rejected, connection alive. + { + let mut stream = TcpStream::connect(&addr).await.unwrap(); + stream + .write_all(&encode_connect_user("testdb", "pw", "ro")) + .await + .unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!(resp[0], 0x02, "readonly user should connect"); + + // Reads succeed. + stream.write_all(&encode_query("User")).await.unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!(resp[0], 0x07, "readonly select should return rows"); + + stream + .write_all(&encode_query("count(User)")) + .await + .unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!(resp[0], 0x08, "readonly count should return scalar"); + + // Every write statement is rejected with a clean permission error. + let writes = [ + r#"insert User { name := "Mallory", age := 1 }"#, + r#"User filter .name = "Alice" update { age := 99 }"#, + r#"User filter .name = "Alice" delete"#, + "drop User", + "alter User add column hacked: str", + "begin", + ]; + for q in writes { + stream.write_all(&encode_query(q)).await.unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!( + resp[0], 0x0A, + "readonly write must be rejected: {q} (got 0x{:02X})", + resp[0] + ); + match Message::decode(&resp).unwrap() { + Message::Error { message } => assert!( + message.contains("permission denied"), + "expected permission-denied error for {q}, got: {message}" + ), + other => panic!("expected Error for {q}, got {other:?}"), + } + } + + // The connection survives the rejections. + stream.write_all(&encode_query("User")).await.unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!(resp[0], 0x07, "connection must stay alive after denials"); + + // And nothing was actually written/dropped. + stream + .write_all(&encode_query("count(User)")) + .await + .unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!(resp[0], 0x08, "table must still exist"); + match Message::decode(&resp).unwrap() { + Message::ResultScalar { value } => { + assert_eq!(value, "1", "row count must be unchanged") + } + other => panic!("expected scalar, got {other:?}"), + } + } + + // Readwrite user keeps full write access. + { + let mut stream = TcpStream::connect(&addr).await.unwrap(); + stream + .write_all(&encode_connect_user("testdb", "pw", "rw")) + .await + .unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!(resp[0], 0x02, "readwrite user should connect"); + + stream + .write_all(&encode_query(r#"insert User { name := "Bob", age := 25 }"#)) + .await + .unwrap(); + let resp = read_response(&mut stream).await; + assert_eq!(resp[0], 0x09, "readwrite insert should succeed"); + } + + handle.abort(); + std::fs::remove_dir_all(&data_dir).ok(); +} diff --git a/crates/storage/src/catalog.rs b/crates/storage/src/catalog.rs index f355f0b..16de6ba 100644 --- a/crates/storage/src/catalog.rs +++ b/crates/storage/src/catalog.rs @@ -8,6 +8,21 @@ use std::io::{self, Read, Write}; use std::path::{Path, PathBuf}; use tracing::{info, warn}; +/// Reject an encoded row that exceeds the single-page capacity BEFORE it is +/// appended to the WAL. The heap performs the same check at its own insert/ +/// update boundary, but the update paths log to the WAL first — a logged +/// record whose row the heap then rejects would poison the next replay. +fn check_encoded_row_size(encoded: &[u8]) -> io::Result<()> { + if encoded.len() > crate::page::MAX_ROW_DATA_SIZE { + return Err(crate::error::StorageError::RowTooLarge { + size: encoded.len(), + max: crate::page::MAX_ROW_DATA_SIZE, + } + .into()); + } + Ok(()) +} + /// Validate that a name (table or column) is safe for use in file paths and /// follows the identifier convention: starts with a letter or underscore, /// followed by letters, digits, or underscores. @@ -935,6 +950,9 @@ impl Catalog { let tbl = self.by_name_mut(table)?; let mut wal_bytes: Vec = Vec::new(); encode_row_into(&tbl.schema, values, &mut wal_bytes); + // Reject oversized rows BEFORE appending the WAL record: a logged + // Update that the heap then rejects would poison the next replay. + check_encoded_row_size(&wal_bytes)?; let tx_id = self.next_tx(); self.wal_log(tx_id, WalRecordType::Update, table, rid, &wal_bytes)?; self.by_name_mut(table)?.update(rid, values) @@ -961,6 +979,8 @@ impl Catalog { let tbl = self.by_name_mut(table)?; let mut wal_bytes: Vec = Vec::new(); encode_row_into(&tbl.schema, values, &mut wal_bytes); + // Same pre-WAL size gate as [`Self::update`]. + check_encoded_row_size(&wal_bytes)?; let tx_id = self.next_tx(); self.wal_log(tx_id, WalRecordType::Update, table, rid, &wal_bytes)?; self.by_name_mut(table)? diff --git a/crates/storage/src/error.rs b/crates/storage/src/error.rs index d49da00..7c9edd9 100644 --- a/crates/storage/src/error.rs +++ b/crates/storage/src/error.rs @@ -26,6 +26,12 @@ pub enum StorageError { #[error("invalid identifier: {0}")] InvalidIdentifier(String), + + /// An encoded row exceeds the single-page capacity. Returned cleanly at + /// the heap insert/update boundary instead of panicking — with + /// `panic = "abort"` a panic here would take down the whole server. + #[error("row too large: {size} bytes exceeds max {max} bytes")] + RowTooLarge { size: usize, max: usize }, } /// Convenience alias used throughout the storage crate. diff --git a/crates/storage/src/heap.rs b/crates/storage/src/heap.rs index 3b3c293..f66e6ef 100644 --- a/crates/storage/src/heap.rs +++ b/crates/storage/src/heap.rs @@ -1,5 +1,6 @@ use crate::disk::DiskManager; -use crate::page::{iter_page_slots, Page, PageType, PAGE_SIZE}; +use crate::error::StorageError; +use crate::page::{iter_page_slots, Page, PageType, MAX_ROW_DATA_SIZE, PAGE_SIZE}; use crate::types::RowId; use rustc_hash::FxHashMap; use std::io; @@ -347,6 +348,24 @@ impl HeapFile { } } + /// Reject rows that can never fit in a single page. This is the clean + /// error boundary for user-supplied oversized values: every mutation + /// entry point (`insert`, `update`) checks BEFORE touching any page so + /// no partial state (e.g. the delete half of a delete+insert update) + /// can land first. With `panic = "abort"` in release builds, the old + /// `expect("row too large for empty page")` was a remote DoS. + #[inline] + fn check_row_size(row_data: &[u8]) -> io::Result<()> { + if row_data.len() > MAX_ROW_DATA_SIZE { + return Err(StorageError::RowTooLarge { + size: row_data.len(), + max: MAX_ROW_DATA_SIZE, + } + .into()); + } + Ok(()) + } + /// Insert encoded row data. Returns RowId. /// /// Mission C Phase 1: uses the hot-page write-back cache. The common @@ -354,6 +373,7 @@ impl HeapFile { /// disk syscalls; the page stays pinned until a different page is /// touched or an explicit flush runs. pub fn insert(&mut self, row_data: &[u8]) -> io::Result { + Self::check_row_size(row_data)?; // WS1 invariant: the persistent mmap covers a snapshot of the file // taken at `enable_mmap()` time (length = num_pages * PAGE_SIZE). // It only becomes unsafe — covering a stale/short region relative @@ -426,7 +446,14 @@ impl HeapFile { // a crash, and that path (`insert_at`) re-initialises a malformed // page before use — so the hot path pays nothing here. let mut page = Page::new(page_id, PageType::Data); - let slot = page.insert(row_data).expect("row too large for empty page"); + // `check_row_size` at the top of `insert` guarantees this fits, but + // keep a graceful error (never a panic) as defence in depth. + let slot = page.insert(row_data).ok_or_else(|| { + io::Error::from(StorageError::RowTooLarge { + size: row_data.len(), + max: MAX_ROW_DATA_SIZE, + }) + })?; if page.free_space() >= 64 { self.pages_with_space.push(page_id); self.mark_free(page_id); @@ -863,6 +890,10 @@ impl HeapFile { /// Mission C Phase 1: in-place updates land on the hot page directly. /// `update_by_filter` and `update_by_pk` both route here. pub fn update(&mut self, rid: RowId, row_data: &[u8]) -> io::Result { + // Reject oversized rows BEFORE the delete+insert fallback below can + // delete the old row — otherwise a failed oversized update would + // destroy the existing data. + Self::check_row_size(row_data)?; self.ensure_hot(rid.page_id)?; { let hot = self.hot_page.as_mut().expect("ensure_hot guarantees Some"); @@ -1807,6 +1838,78 @@ mod tests { std::fs::remove_file(&path).ok(); } + #[test] + fn test_oversized_insert_returns_error_and_heap_survives() { + let (mut heap, path) = temp_heap("oversized_insert"); + let schema = user_schema(); + + // A row larger than any empty page can hold must be a clean error, + // not a panic (panic = "abort" kills the whole server process). + let big = vec![0xABu8; PAGE_SIZE]; + let err = heap.insert(&big).unwrap_err(); + assert!( + err.to_string().contains("row too large"), + "unexpected error: {err}" + ); + + // The heap must remain fully usable afterwards. + let rid = heap + .insert(&encode_row( + &schema, + &[Value::Str("ok".into()), Value::Int(1)], + )) + .unwrap(); + assert!(heap.get(rid).is_some()); + assert_eq!(heap.scan().count(), 1); + drop(heap); + std::fs::remove_file(&path).ok(); + } + + #[test] + fn test_insert_at_max_row_size_succeeds() { + use crate::page::MAX_ROW_DATA_SIZE; + let (mut heap, path) = temp_heap("max_row"); + // Exactly the max must still fit on a fresh page. + let exact = vec![0x42u8; MAX_ROW_DATA_SIZE]; + let rid = heap.insert(&exact).unwrap(); + assert_eq!(heap.get(rid).unwrap().len(), MAX_ROW_DATA_SIZE); + // One byte more must be rejected. + let over = vec![0x42u8; MAX_ROW_DATA_SIZE + 1]; + let err = heap.insert(&over).unwrap_err(); + assert!( + err.to_string().contains("row too large"), + "unexpected error: {err}" + ); + drop(heap); + std::fs::remove_file(&path).ok(); + } + + #[test] + fn test_oversized_update_returns_error_and_row_intact() { + let (mut heap, path) = temp_heap("oversized_update"); + let schema = user_schema(); + let rid = heap + .insert(&encode_row( + &schema, + &[Value::Str("Alice".into()), Value::Int(30)], + )) + .unwrap(); + let old_bytes = heap.get(rid).unwrap(); + + // Updating to an oversized row must fail cleanly WITHOUT deleting + // the old row (the delete+insert fallback must not fire). + let big = vec![0xCDu8; PAGE_SIZE]; + let err = heap.update(rid, &big).unwrap_err(); + assert!( + err.to_string().contains("row too large"), + "unexpected error: {err}" + ); + assert_eq!(heap.get(rid).unwrap(), old_bytes, "old row must survive"); + assert_eq!(heap.scan().count(), 1); + drop(heap); + std::fs::remove_file(&path).ok(); + } + #[test] fn test_multi_page_span() { let (mut heap, path) = temp_heap("multipage"); diff --git a/crates/storage/src/page.rs b/crates/storage/src/page.rs index 114f444..2f898e5 100644 --- a/crates/storage/src/page.rs +++ b/crates/storage/src/page.rs @@ -25,6 +25,15 @@ const SLOT_COUNT_SIZE: usize = 2; // u16 at bottom of page const SLOT_ENTRY_SIZE: usize = 4; // u16 offset + u16 length per slot const DELETED_MARKER: u16 = 0xFFFF; +/// Maximum encoded row size that can ever fit in a single page: a fresh +/// empty page minus the slot-count word and the row's own slot entry. +/// Anything larger must be rejected at the heap boundary as a clean +/// `StorageError::RowTooLarge` — with `panic = "abort"` in release builds, +/// a panicking insert path would otherwise kill the whole server process +/// (remote DoS via one oversized `insert`). +pub const MAX_ROW_DATA_SIZE: usize = + PAGE_SIZE - PAGE_HEADER_SIZE - SLOT_COUNT_SIZE - SLOT_ENTRY_SIZE; + /// Byte range holding the page CRC32 (WS3). Lives just after the legacy /// 16-byte header so the slot directory at the bottom of the page is /// untouched — old and new pages share the same slot/slot_count layout, diff --git a/SMOKE-AUDIT.md b/docs/audits/2026-05-26-smoke-audit.md similarity index 96% rename from SMOKE-AUDIT.md rename to docs/audits/2026-05-26-smoke-audit.md index bf170af..8f8c11a 100644 --- a/SMOKE-AUDIT.md +++ b/docs/audits/2026-05-26-smoke-audit.md @@ -1,13 +1,14 @@ # PowDB Smoke Audit — 2026-05-26 -> **HISTORICAL — superseded as of v0.4.4 (2026-06-05).** This is a point-in-time +> **HISTORICAL — superseded (current release: v0.4.5).** This is a point-in-time > smoke audit of the pre-release 0.4.0 build. Nearly every finding has since been > resolved: ROLLBACK now undoes heap writes (verified by transaction tests in -> `crates/query/src/executor/tests.rs`), the CHANGELOG has entries through 0.4.4, -> all four crates are published on crates.io at 0.4.4, version pins and the MSRV -> are current, and issue/PR templates exist. The three later durability P0s that -> 0.4.4 fixed are not in this audit — see `CHANGELOG.md`. Kept unaltered for -> provenance; do not treat its grades or status claims as current. +> `crates/query/src/executor/tests.rs`), the CHANGELOG has entries through 0.4.5, +> all six crates (including the new powdb-auth and powdb-backup) are published on +> crates.io at 0.4.5, version pins and the MSRV are current, and issue/PR +> templates exist. The three later durability P0s that 0.4.4 fixed are not in +> this audit — see `CHANGELOG.md`. Kept unaltered for provenance; do not treat +> its grades or status claims as current. ## Executive Summary @@ -296,7 +297,7 @@ ## Screenshots -All 12 screenshots saved to `screenshots/`: +12 screenshots were captured during the audit to a local `screenshots/` directory (gitignored; not retained in the repo): - `github-pages-homepage.png` — Landing page (polished, professional) - `github-repo.png` — Repository page (good metadata) - `github-org.png` — User profile page diff --git a/docs/benchmarks/2026-06-09-local-apple-silicon.md b/docs/benchmarks/2026-06-09-local-apple-silicon.md new file mode 100644 index 0000000..2b53aeb --- /dev/null +++ b/docs/benchmarks/2026-06-09-local-apple-silicon.md @@ -0,0 +1,53 @@ +# Local bench snapshot — 2026-06-09 (Apple Silicon) + +Captured from `crates/compare/results.csv` (2026-06-09, macOS arm64, release +build). This file preserves the run because `results.csv` is in `.gitignore` +(intentionally: it's a bench artifact, rewritten on every run). + +> **These are local laptop numbers, NOT the CI regression baseline.** +> `crates/bench/baseline/main.json` is captured on CI hardware (GitHub +> ubuntu x86) and must never be compared against — or rebaselined from — +> arm64 laptop runs. + +## Methodology + +- Harness: `cargo run --release -p powdb-compare` +- Fixture: 100,000 rows on each engine, identical schema +- Engines: PowDB (in-process, `WalSyncMode::Off`) + SQLite (in-process, + `:memory:`) — both entirely in RAM +- Each workload: ns/op reported as median + +## Results + +| workload | PowDB | SQLite | ratio (SQLite÷PowDB) | +| --- | ---: | ---: | ---: | +| point_lookup_indexed | 54 ns | 190 ns | 3.5x | +| point_lookup_nonindexed | 239.5 us | 301.9 us | 1.3x | +| scan_filter_count | 233.4 us | 1.31 ms | 5.6x | +| scan_filter_project_top100 | 6.4 us | 8.2 us | 1.3x | +| scan_filter_sort_limit10 | 2.10 ms | 6.02 ms | 2.9x | +| agg_sum | 187.7 us | 1.40 ms | 7.5x | +| agg_avg | 280.2 us | 1.65 ms | 5.9x | +| agg_min | 137.0 us | 1.60 ms | 11.7x | +| agg_max | 142.6 us | 1.38 ms | 9.7x | +| multi_col_and_filter | 1.36 ms | 3.05 ms | 2.2x | +| insert_single | 297 ns | 627 ns | 2.1x | +| insert_batch_1k | 141 ns | 203 ns | 1.4x | +| update_by_pk | 42 ns | 261 ns | 6.3x | +| update_by_filter | 1.74 ms | 4.42 ms | 2.5x | +| delete_by_filter | 1.18 ms | 1.59 ms | 1.3x | + +**Score: PowDB faster on all 15 workloads on this run.** Contrast with the +[2026-04-07 snapshot](2026-04-07-wide-bench-snapshot.md) (5 wins, 10 losses) +— the write-path and point-lookup losses identified there have since been +closed. + +## Reproducibility + +```bash +cargo run --release -p powdb-compare +# rewrites crates/compare/results.csv +``` + +Re-running will produce slightly different absolute numbers (±10-15% on an +idle laptop), but the verdicts are stable. diff --git a/docs/design/powdb-implementation-brief.md b/docs/design/powdb-implementation-brief.md index dd9b09c..6b97146 100644 --- a/docs/design/powdb-implementation-brief.md +++ b/docs/design/powdb-implementation-brief.md @@ -1,5 +1,7 @@ # PowDB: Complete implementation brief +> **Historical design doc (pre-implementation).** For current architecture see `CLAUDE.md` / `AGENTS.md`. + This document contains everything needed to implement PowDB from scratch. It is the single source of truth — all architectural decisions are backed by production benchmarks and explained with rationale. diff --git a/docs/design/powdb-wire-protocol.md b/docs/design/powdb-wire-protocol.md index 159d132..1e38992 100644 --- a/docs/design/powdb-wire-protocol.md +++ b/docs/design/powdb-wire-protocol.md @@ -1,5 +1,7 @@ # PowDB: Wire protocol and engine architecture +> **Historical design doc (pre-implementation).** For current architecture see `CLAUDE.md` / `AGENTS.md`. + ## Engine architecture PowDB is a library first, server second. The core engine is a single library diff --git a/docs/getting-started.md b/docs/getting-started.md index 7bd9e05..0f46726 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -44,7 +44,7 @@ cargo run --release -p powdb-cli You should see: ``` -PowDB v0.4.4 — embedded mode +PowDB v0.4.5 — embedded mode Data directory: ./powdb_data Type PowQL queries. Use Ctrl-D to exit. @@ -119,6 +119,8 @@ powql> insert User { name := "Grace", email := "grace@example.com" } 1 row affected ``` +> **Multi-row insert:** you can also insert many rows in one statement by separating row blocks with commas -- `insert User { ... }, { ... }, { ... }`. One statement means one WAL fsync and one network round trip, and validation is all-or-nothing. See [INSERT in the PowQL reference](POWQL.md#insert). + > **Note:** Each autocommit `insert` fsyncs to the write-ahead log for durability, which caps single-row inserts at roughly a few hundred per second on real disks. For bulk loads, wrap many inserts in a `begin` / `commit` transaction -- they share a single fsync at commit and run dozens of times faster, still fully durable. See [Transactions](POWQL.md#transactions). --- @@ -443,15 +445,15 @@ cargo run --release -p powdb-cli -- --remote localhost:5433 Output: ``` -PowDB v0.4.4 — remote mode +PowDB v0.4.5 — remote mode Connecting to localhost:5433 ... -Connected to db `main` (server v0.4.4) +Connected to db `main` (server v0.4.5) Type PowQL queries. Use Ctrl-D to exit. powql> ``` -From here, the same PowQL statements work as embedded mode. Some status messages are normalized by the wire protocol -- for example, DDL statements return an affected-row status such as `0 rows affected` in remote mode instead of the embedded REPL's `type User created` message. The server handles concurrent readers and uses a write-ahead log for durability. +From here, the same PowQL statements work as embedded mode -- DDL statements return the same friendly status messages too (e.g. `type User created`). The server handles concurrent readers and uses a write-ahead log for durability. ### Password authentication @@ -532,3 +534,5 @@ relying on the bootstrap env vars. This tutorial covered the basics: tables, inserts, queries, aggregates, updates, indexes, and deletes. PowDB supports much more, including joins, group by, subqueries, materialized views, and set operations. See the full language reference: [PowQL Reference](POWQL.md) + +Running in production? Set up backups: [Backup & restore](backup-and-restore.md) covers full and incremental backups plus coarse point-in-time recovery (offline -- stop the server first). diff --git a/docs/gtm-strategy.md b/docs/gtm-strategy.md new file mode 100644 index 0000000..84766c8 --- /dev/null +++ b/docs/gtm-strategy.md @@ -0,0 +1,191 @@ +# PowDB Go-To-Market Strategy + +**Author:** Product Lead (strategist mode) +**Date:** 2026-06-09 +**Version reviewed:** v0.4.5 (crates.io, npm, ghcr, GitHub Pages) +**Status of the product:** Real engine, honest benchmarks, pre-1.0, single-node, tiny team (ZVN/Kirby). + +--- + +## The Take (read this first) + +PowDB is a genuinely fast, genuinely pure-Rust embedded engine with an honest 3–10x SQLite win on the workloads it's built for. That is real and defensible at the *engine* level. But the headline pitch — "faster than SQLite" — is a **losing wedge**, because (a) SQLite is "fast enough" for ~95% of the people who pick it, and (b) the moment performance is the axis, **Turso Database (the from-scratch Rust SQLite rewrite, beta as of May 2026) eats your exact positioning while keeping SQL compatibility.** Same "pure Rust, no C, from scratch" story, but a user keeps every ORM, driver, and tool. On a pure perf-vs-SQLite axis, PowQL is a tax with no offsetting reward for most teams. + +The only defensible wedge is the one the brief already half-identifies: **agent-native, in-process state for Rust AI-agent runtimes.** Not "database for AI agents" in the cloud-memory-layer sense (Oracle, TiDB, mem0, Zep own that and it's a vector/graph game). The wedge is narrower and more honest: *the embedded query engine that a Rust agent harness reaches for to store and query its own structured state, where PowQL's AST=plan-tree design and the AGENTS.md cheat-sheet make small models reliably generate correct queries in-context.* That is a real gap and nobody is sitting in it. Everything else — edge analytics, generic Rust embedding — is a fast-follower fight you will lose on ecosystem. + +**My recommendation:** Reposition from "faster SQLite" to **"the embeddable query engine built for agents and the Rust services that run them."** Lead with DX and agent-correctness, keep the perf numbers as proof-of-craft, not as the headline. Pick ONE launch wedge (Rust agent-state) and go deep. Be brutally honest in the README about where SQLite/Turso win — you already are, and that honesty is itself a marketing asset in this audience. + +--- + +## 1. Positioning Statement + One-Line Pitch + +**Positioning statement:** +> For Rust developers building AI agents and latency-sensitive services who need to store and query structured state in-process, PowDB is a pure-Rust embedded database whose pipeline query language (PowQL) is designed so a small language model can generate correct queries from a one-page cheat sheet — and whose compiled execution engine runs scan and aggregate workloads 3–10x faster than SQLite. Unlike SQLite or Turso, PowDB removes the SQL translation tier entirely; unlike cloud agent-memory platforms, it runs embedded with zero network hop. + +**One-line pitch (lead with this):** +> **PowDB — the pure-Rust embedded query engine your AI agent can actually drive.** + +**Backup one-liners by audience:** +- Rust perf crowd: *"A from-scratch Rust database where the parser's AST is the query plan — 3–10x SQLite on scans and aggregates, zero C in the build."* +- Agent builders: *"Give your agent a database it can query correctly from a one-page prompt, in-process, no SQL-injection guesswork."* + +**What to STOP saying:** "3–10x faster than SQLite" as the *first* line. It invites the one comparison you lose strategically (Turso) and the one you can't win culturally (SQLite is good enough). Demote it to proof, not thesis. + +--- + +## 2. Honest Competitive Matrix + +Legend: ✅ genuine strength · ⚠️ partial / caveated · ❌ genuine weakness · — n/a + +| Dimension | **PowDB** | SQLite | DuckDB | Turso DB (libSQL + Limbo rewrite) | redb | +|---|---|---|---|---|---| +| Query language | PowQL (custom pipeline) ⚠️ | SQL ✅ | SQL ✅ | SQL ✅ | none (KV) ❌ | +| Pure Rust, no C | ✅ | ❌ (C) | ❌ (C++) | ✅ (Limbo) / ❌ (libSQL is C) | ✅ | +| Scan/aggregate perf | ✅ 3–10x SQLite | baseline | ✅✅ (columnar, beats both) | ≈ SQLite | n/a (KV) | +| OLTP point writes/lookups | ⚠️ competitive | ✅ mature | ⚠️ weak at OLTP | ✅ | ✅ | +| Maturity / battle-testing | ❌ pre-1.0 | ✅✅ billions | ✅ mature | ⚠️ libSQL prod, rewrite beta | ✅ 1.0, stable format | +| Tooling/ecosystem (ORMs, BI, drivers) | ❌ TS + Rust client only | ✅✅ everything | ✅ growing | ✅ (SQLite-compatible) | ❌ minimal | +| Edge/replication/sync | ❌ single-node | ⚠️ via libSQL | ❌ | ✅✅ embedded replicas, edge sync | ❌ | +| Server mode + auth + TLS | ✅ (shared pw / users) | ❌ not in core | ⚠️ Quack ext (new) | ✅ managed cloud | ❌ | +| Durability (WAL, crash recovery) | ✅ | ✅ | ✅ | ✅ | ✅ | +| Transactions | ✅ begin/commit/rollback | ✅ | ✅ | ✅ | ✅ | +| Vector / RAG search | ❌ | ⚠️ ext | ⚠️ ext | ✅ DiskANN native | ❌ | +| MVCC / concurrent writers | ❌ single-writer | ⚠️ WAL readers | ⚠️ | ✅ | ✅ MVCC readers | +| Agent-driveable query lang | ✅ (design intent + AGENTS.md) | ⚠️ (LLMs know SQL ~95%) | ⚠️ (SQL) | ⚠️ (SQL) | ❌ | +| LLM training-data coverage of the lang | ❌ ~zero | ✅✅ massive | ✅ | ✅ | n/a | + +### Where PowDB genuinely WINS today +1. **Pure-Rust + no C toolchain + fast aggregates, all at once.** No other option gives you all three. DuckDB is faster on analytics but is C++ and OLAP-shaped. Turso's Rust rewrite is SQL but still beta and not aggregate-optimized the way PowDB is. +2. **Smallest readable codebase.** ~20K lines, no generated parser, no bytecode VM. For a Rust shop that wants to *own and audit* its storage layer, that's a real value prop redb shares but SQLite/DuckDB/Turso don't. +3. **Embedded server mode with auth + TLS out of the box.** SQLite needs extensions; redb is KV-only. PowDB ships a TCP server today. +4. **Design coherence for in-context query generation.** AST=plan, one-page AGENTS.md, lowercase keywords, dot-field syntax — *if* the agent thesis holds, this is the one place PowDB is purpose-built and nobody else is. + +### Where PowDB genuinely LOSES today (say this out loud) +1. **The query language has no ecosystem and no LLM prior.** This is the single biggest liability. Every team that picks PowDB rewrites every query, every model has to be taught PowQL in-context every time, and no ORM/BI tool will ever speak it. SQL's 94–95% LLM execution accuracy comes from billions of training examples PowQL will never have. +2. **Turso owns "Rust + from-scratch + faster" with SQL compat.** Your differentiator-of-record (pure Rust, no SQL parse tax) is shared by a VC-funded team that kept SQL. That makes "pure Rust" table stakes, not a moat. +3. **DuckDB wins analytics outright.** If the use case is genuinely scan/aggregate-heavy, a serious evaluator benchmarks DuckDB and PowDB loses on columnar workloads. Your bench is vs SQLite, not vs the actual analytics leader. +4. **Maturity gap is existential for a DB.** Pre-1.0, shifting on-disk format, 3 fuzz targets vs SQLite's OSS-Fuzz corpus. Nobody puts their source-of-truth data on a v0.4 single-team engine. This caps you to *derived/ephemeral* state, which is actually fine — see the wedge. +5. **No MVCC, single writer.** Fine for embedded single-agent, disqualifying for any multi-writer service. + +**Honest moat assessment:** The engine moat (compiled predicates + pure Rust) is *thin and shrinking* — Turso closes it. The only **expanding** moat is **PowQL-as-agent-interface coherence + the AGENTS.md DX pattern**, and that moat only exists if you invest in it deliberately and prove it. Right now it's a hypothesis, not a moat. + +--- + +## 3. Unique Use Cases, Ranked by Wedge Potential + +### Wedge #1 (GO HERE): In-process state store for Rust AI-agent runtimes +**The job:** A Rust agent harness (think a Carl-Code/opencode-style loop, a tool-using agent, a workflow engine) needs to persist and query structured state — task queues, tool-call history, episodic facts, scratchpad tables — and let the *model itself* read/write that state via generated queries. + +**Why incumbents serve it poorly:** +- Cloud agent-memory (Oracle, TiDB, mem0, Zep) is network-hopped, vector/graph-shaped, and overkill for in-loop structured scratch state. Wrong latency class, wrong shape. +- SQLite works but the agent generates SQL against an unknown schema and you're back to error-retry loops and injection-escaping (PowDB's own TS client has no param binding yet — note this gap). +- redb is KV; the agent can't express `filter .status = "pending" order .created desc limit 5`. + +**PowDB's unfair advantage:** In-process (zero hop), pure-Rust (drops straight into the harness binary), and PowQL + AGENTS.md is a *deliberately small grammar a 7B–20B model can be taught in one prompt.* The AST=plan property means a generated query is either parseable-and-runnable or a clean parse error you feed back — exactly the "self-healing retry" loop the 2026 text-to-SQL literature describes, but over a grammar small enough to fully specify in context. + +**Wedge size & expansion:** Small today (Rust agent harnesses are a niche of a niche), but it's the fastest-growing developer category in 2026 and ZVN is *literally building one*. Land here, expand to "embedded state for any latency-sensitive Rust service." + +**Falsifiable:** If a 14B model with AGENTS.md in context can't hit >90% valid-query generation on a 10-table schema, the wedge is dead. **Test this in week one.** + +### Wedge #2: Embedded analytics inside a Rust service (the "hot rollup" cache) +**The job:** A Rust web service doing per-request aggregates (dashboards, counters, leaderboards, top-N) over tables that fit on disk, where the aggregate is on the request hot path. + +**Why incumbents serve it poorly:** SQLite is 3–10x slower here (your bench proves it); DuckDB is C++ and OLAP-process-shaped, not a great per-request embed; Turso isn't aggregate-tuned. + +**Why it's #2 not #1:** Real but small win, and DuckDB-as-library is a credible counter for anyone analytics-serious. The PowQL tax is hardest to justify here because these are developer-authored queries, not agent-generated — so the "small grammar" advantage doesn't apply. + +### Wedge #3: Pure-Rust / no-C constrained build targets +**The job:** Wasm-adjacent, minimal containers, supply-chain-paranoid shops that refuse `bindgen`/C in the build. + +**Why incumbents serve it poorly:** SQLite/DuckDB are C/C++. redb is the real competitor here but is KV-only — PowDB offers a query language. + +**Why it's #3:** Genuine but tiny audience, and Turso's Rust rewrite erodes it. Good *secondary* talking point, not a launch wedge. + +### Wedge #4 (DO NOT LEAD WITH): "Database for AI agents" (the cloud memory-layer meaning) +This is a crowded, well-funded category (Oracle AI Agent Memory, TiDB, mem0, Zep, Databricks Lakebase) playing a vector + temporal-graph + horizontal-scale game. PowDB has no vectors, no scale-out, no memory abstractions. **Do not position here** — you'll be measured against features you don't have and lose instantly. Use Wedge #1's *narrower, in-process, structured-state* framing instead and explicitly distinguish it from "agent memory layer." + +--- + +## 4. Target Personas + Channels + +### Primary persona — "Harness Hannah," the Rust agent-runtime builder +- Builds an agent loop / tool-runner / workflow engine in Rust. Cares about latency, binary size, no-C builds, and *letting the model drive tools reliably.* +- Reads: this-week-in-rust, lobste.rs, r/rust, HN, the agent-builder Discords/X. +- Painkiller: "my agent generates broken queries against SQLite and I babysit the retry loop." PowQL's small grammar + AGENTS.md is the pitch. +- **This is the persona ZVN already is.** Dogfood PowDB inside the ZVN agent stack and write that up — it's the single most credible artifact you can ship. + +### Secondary persona — "Perf-Rust Pete," the latency-sensitive Rust service dev +- Has a hot-path aggregate, already on tokio, won't add a C dep. Wants `cargo install` and 3x. +- Reads: same Rust channels + benchmark threads. Motivated by the honest comparison doc. + +### Tertiary — "Skeptical Sam," the senior eng / DB nerd on HN/lobste.rs +- Won't adopt, but will *amplify or destroy* your launch. Wins respect through honesty (your vs-SQLite doc is already calibrated for him), loses instantly to overclaiming. Treat the HN thread as the real product surface. + +### Channels (ranked) +1. **lobste.rs + r/rust + this-week-in-rust** — your actual buyers live here. Highest signal, lowest cost. +2. **A killer "show, don't tell" dogfood writeup** — "We let a 14B model run our agent's state store with a one-page cheat sheet. Here's the eval." This is your wedge proof AND your best content all at once. +3. **HN Show HN** — high-variance, high-reward. Only after the dogfood eval exists, because the top comment *will* be "why not Turso/SQLite?" and you need a crisp, honest, data-backed answer ready. +4. **Agent-builder communities (X/Discord)** — narrower but exactly Wedge #1. Seed via the eval writeup. +5. **docs.rs + crates.io quality** — passive but compounding; Rust devs judge by docs.rs polish. + +### Content strategy (3 anchor pieces, not a blog mill) +1. **"Can a small model drive a database? An eval of LLM-generated PowQL vs SQL."** — the thesis-defining/falsifying artifact. Publish the numbers even if they're mixed; honesty *is* the brand. +2. **"PowDB vs SQLite vs DuckDB vs Turso: when each one is right" (extend the existing honest doc to 4-way).** Ranks for the comparison searches and earns trust by recommending competitors when they're right. +3. **"How we built a query language whose AST is its plan tree"** — engine-craft piece for the perf/DB nerds; this is the "read the code" crowd magnet. + +--- + +## 5. Launch Plan — 30 / 60 / 90 Days (sized for 1–2 people + agents) + +### Next 30 days — **Prove or kill the wedge. No public launch yet.** +1. **Run the falsification eval (Wedge #1).** Build a harness: 10-table schema, 50–100 natural-language tasks, AGENTS.md in context, a 14B-class model (use the ZVN inference gateway), measure valid-query + correct-result rate vs the same model writing SQLite SQL. **This single number decides the whole strategy.** Track parse-success, execution-success, semantic-correctness separately. +2. **Close the one disqualifying DX gap: parameter binding over the wire.** AGENTS.md admits "no parameter binding yet" and the agent wedge *requires* safe value insertion. Without it, you're telling agent builders to escape strings by hand. Ship prepared-statement placeholders in the TS + Rust clients. (This is the highest-leverage feature on the roadmap for the chosen wedge.) +3. **Write the 4-way honest comparison doc** (add DuckDB + Turso to the existing SQLite doc). Needed before any HN exposure. +4. **Dogfood PowDB in the ZVN agent stack** for one real state-store use case. Even one shipped internal use is worth more than any benchmark. + +**Gate:** if the eval shows PowQL generation is materially *worse* than SQL and AGENTS.md can't close the gap, **stop and pivot the positioning to pure perf/Rust-embed (Wedge #2/#3) and lower ambitions.** Do not launch the agent narrative on a falsified thesis. + +### 30–60 days — **Soft launch to the friendly Rust audience.** +5. **Publish the eval writeup** + the 4-way comparison. Post to lobste.rs and r/rust (NOT HN yet). Gather the objections — they're your roadmap. +6. **Ship `powdb-agent` examples crate:** a runnable example of an LLM driving PowDB via AGENTS.md (works with the ZVN gateway and OpenAI/Anthropic). This is the wedge made tangible. +7. **Harden the honesty surface:** make sure README's top line matches the new positioning (lead with DX/agent, demote the bench headline). Tighten getting-started so `cargo install` → first query is under 2 minutes. +8. **Cut a 0.5.0** that bundles param-binding + any eval-driven fixes. Signal momentum. + +### 60–90 days — **The HN swing + decide on 1.0 path.** +9. **Show HN**, only with: the eval, the 4-way doc, the agent example, and a maintainer (Kirby) ready to answer "why not Turso/SQLite/DuckDB?" live for 6 hours. The honest comparison docs are your armor. +10. **Publish the on-disk format stability commitment + 1.0 roadmap.** The maturity objection is the #2 adoption blocker; a credible "format frozen at 1.0, here's the date" calms it. +11. **Instrument adoption:** crates.io downloads by crate, GitHub stars velocity, *and the one metric that matters* — number of external repos that depend on `powdb-*` and use it for agent state (search GitHub). One real external agent-builder adopter > 1,000 stars. +12. **Decide:** double down on agent wedge (if eval + adoption signal yes) or settle into "the fast pure-Rust embedded engine" niche (if the agent thesis underdelivers). + +--- + +## 6. Risks, Thesis-Falsifiers, and Metrics + +### What would FALSIFY the core thesis (and what to do) +| Thesis | Falsifier | If falsified | +|---|---|---| +| PowQL is easier for small models to generate correctly than SQL | 14B model with AGENTS.md scores ≤ SQL on valid+correct query rate | Kill the agent positioning; fall back to perf/Rust-embed niche | +| Removing the SQL tier is a durable perf moat | Turso's Rust rewrite hits GA and matches/beats PowDB while keeping SQL | Stop competing on perf-vs-SQLite; compete only on DX/agent + readability | +| There's demand for a non-SQL embedded DB | <5 external repos adopt in 90 days despite a clean launch | Reframe as a library/teaching engine; lower roadmap ambition, keep it as ZVN-internal infra | +| Aggregate perf is the buying reason | Evaluators benchmark DuckDB and leave | Concede analytics to DuckDB; own "fast *transactional* embedded for Rust services + agents" | + +### Top risks +1. **The PowQL ecosystem tax is unwinnable at scale.** *Mitigation:* never fight on ecosystem; win only where the query is agent-generated-in-context (so no human ecosystem is needed) or developer-owned-and-tiny. +2. **Turso commoditizes "pure Rust from scratch."** *Mitigation:* shift the moat to DX/agent-coherence + auditable small codebase; treat pure-Rust as table stakes. +3. **Maturity blocks all serious adoption.** *Mitigation:* position for *derived/ephemeral* state (agent scratch, hot caches, rollups) where data loss is recoverable; publish a format-freeze 1.0 commitment. +4. **Tiny team can't sustain a DB.** *Mitigation:* keep scope brutally small (single-node, embedded, the wedge); resist every "add replication/vectors/UDFs" request that isn't the wedge. Use the multi-agent dev workflow ZVN already runs. +5. **Launch overclaim torches credibility.** *Mitigation:* the existing honesty discipline is your single best asset — protect it. Every benchmark claim must be reproducible (`cargo run -p powdb-compare`) before it's public. + +### Metrics to track (rollups, not vanity) +**North-star:** # of external repositories using `powdb-*` for agent/service state (not stars). This is the only metric that proves the wedge. +- **Wedge-proof:** LLM-generated-PowQL valid+correct rate (the eval number), tracked per model size. +- **Activation:** time-to-first-query from `cargo install` (target < 2 min); % of new repos that run a second session (retention proxy). +- **Funnel:** crates.io downloads per crate, docs.rs visits, getting-started → second-query drop-off. +- **Credibility:** HN/lobste.rs sentiment (does the top comment defend or dismiss?), reproducibility complaints (target: zero). +- **Anti-metric to ignore:** raw GitHub stars. Stars from a benchmark HN post don't equal adoption and will mislead the roadmap. + +--- + +## Bottom line for the team + +You built a real engine with real, honestly-measured wins. The trap is letting "faster than SQLite" be the story — it's the one fight where SQLite is good enough and Turso out-positions you. The prize is being **the embedded query engine that an AI agent can correctly drive in-process**, proven first inside ZVN's own agent stack. Run the eval in week one; it tells you whether you have a category or a faster SQLite. Either way, your radical honesty in the docs is the most valuable thing you've shipped — don't trade it for a louder headline. diff --git a/docs/powdb-vs-sqlite.md b/docs/powdb-vs-sqlite.md index 94ec5a2..f3766b9 100644 --- a/docs/powdb-vs-sqlite.md +++ b/docs/powdb-vs-sqlite.md @@ -47,8 +47,11 @@ between the two. - **You're already shipping the C toolchain.** If your build already compiles `aws-lc`, `openssl`, or any other C dep, the `libsqlite3-sys` cost is zero. -- **You want full MVCC, RBAC, online backups, or any of the - decade-of-features SQLite has.** PowDB has none of these yet. +- **You want full MVCC, online backups, or any of the decade-of-features + SQLite has.** PowDB 0.4.5 shipped role-based users (admin / readwrite / + readonly) and offline full/incremental backup with coarse point-in-time + recovery, but there is still no MVCC and no *online* backup -- backups + require stopping the server. ## Side-by-side feature table @@ -67,6 +70,7 @@ between the two. | Server mode | Yes (binary wire protocol, TLS, auth) | Not in core (extensions exist) | | Fuzz testing | 3 cargo-fuzz targets (lexer, parser, roundtrip) | OSS-Fuzz, decades of corpora | | Crash recovery | WAL replay + page-zero recovery + index rebuild | WAL/rollback journal | +| Backup | Offline full/incremental + coarse PITR (0.4.5) | Online backup API, `.backup`, VACUUM INTO | | On-disk format stability | Pre-1.0, may shift | Stable for decades | | Production deployments | Pre-1.0 | Billions | @@ -134,7 +138,7 @@ Results land in `crates/compare/results.csv`. ## Caveats and roadmap - **PowDB is pre-1.0.** The on-disk format may shift across minor versions. - Pin a version (`cargo install powdb-cli --version 0.4.4 --locked`) and + Pin a version (`cargo install powdb-cli --version 0.4.5 --locked`) and expect to re-bench / re-import on upgrades until 1.0. - **SQLite is the safe default.** Decades of production exposure, an enormous test suite, and tools everywhere. If you're not sure, you diff --git a/clients/ts/SPRINT-PLAN.md b/docs/superpowers/plans/2026-04-16-ts-client-v02-sprint.md similarity index 98% rename from clients/ts/SPRINT-PLAN.md rename to docs/superpowers/plans/2026-04-16-ts-client-v02-sprint.md index f2d6e90..f1911bf 100644 --- a/clients/ts/SPRINT-PLAN.md +++ b/docs/superpowers/plans/2026-04-16-ts-client-v02-sprint.md @@ -1,4 +1,7 @@ # Sprint Plan — PowDB TS Driver v0.2 + +> **Completed / historical.** This sprint shipped in client 0.3.x; kept for provenance. + Generated: 2026-04-16 Based on: conversation review of `clients/ts/` (see session thread) diff --git a/docs/superpowers/plans/2026-06-09-easy-wins-sprint.md b/docs/superpowers/plans/2026-06-09-easy-wins-sprint.md new file mode 100644 index 0000000..9ec0180 --- /dev/null +++ b/docs/superpowers/plans/2026-06-09-easy-wins-sprint.md @@ -0,0 +1,827 @@ +# PowDB Easy-Wins Sprint Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship six high-leverage improvements: truthful EXPLAIN, real B+tree range scans, UNIQUE constraints, wire-level parameter binding, multi-line REPL input, and a repeatable agent-DX eval harness — each landing green and separately committed. + +**Architecture:** PowQL pipeline is lexer → parser → planner (pure, no catalog) → executor (lowers speculative plans at runtime against the catalog). Storage is heap + B+tree + WAL + catalog in `crates/storage`. Server is a length-prefixed binary TCP protocol in `crates/server`. TS client in `clients/ts`. + +**Tech Stack:** Rust workspace, criterion bench, TS client (Node 22, pnpm). + +**Branch:** All work happens on a new branch `feat/easy-wins-sprint`, cut from main after PR #81 merges. Never push to `main`. Never touch `crates/bench/baseline/main.json` (CI-hardware only). + +```bash +cd /Users/macbookpro-kirby/Desktop/Coding/ZVN/PowDB +git checkout main && git pull +git checkout -b feat/easy-wins-sprint +``` + +**Per-task gate (applies to every task, referenced as "GATE" below):** +```bash +cargo test --workspace +cargo clippy --workspace --all-targets -- -D warnings +cargo fmt --all +``` + +**Key reconnaissance facts the implementer must know (verified against source):** +- `lower_unindexed_range_scans` (`crates/query/src/executor/plan_exec.rs:3372`) ALREADY recurses into `PlanNode::Explain` (line 3462) and already lowers unindexed `RangeScan` → `Filter(SeqScan)`. The actual EXPLAIN lie is `IndexScan`: the lowering pass treats it as a leaf (falls through `_ => plan.clone()` at 3477), while the executor's `IndexScan` arm (plan_exec.rs:1588) silently falls back to a compiled-predicate scan at 1611–1636 when `!tbl.has_index(column)`. +- `RangeScan` execution ALREADY walks the btree — but only for unique indexes (`plan_exec.rs:1687`, gated by `tbl.is_index_unique(column) == Some(true)`; same gate in the lowering pass at 3385). Non-unique indexes store composite keys `(col_val, rid)` (`crates/storage/src/btree.rs:754`, `make_composite_key`) with order-preserving big-endian encodings, plus `make_prefix_start`/`make_prefix_end`/`rid_from_composite` helpers (btree.rs:800–892). `alter T add index .c` creates NON-unique indexes (`plan_exec.rs:1451` → `Catalog::create_index` → `create_index_unique(…, false)`). +- Uniqueness infrastructure half-exists: `IndexedCol.unique` (`crates/storage/src/table.rs:35`), `IndexedColMeta { name, unique }` persisted by `Catalog::persist()` (table.rs:177–185, catalog.rs:1173–1186), `Table::create_index_with_unique` (table.rs:1012). But unique-index insert OVERWRITES duplicates ("correct for PKs", table.rs:405–413) — no rejection anywhere. Nulls are never indexed (table.rs:402). +- Prepared machinery: `Engine::prepare` / `execute_prepared(prep, &[Literal])` (`crates/query/src/executor/prepared.rs:98,260`). `Literal` has NO Null variant (`ast.rs:322`), but `Token::Null` exists (`token.rs:64`) — token-level substitution sidesteps that gap. +- Protocol msg ids `0x04–0x06` are free (`crates/server/src/protocol.rs:4-14`). Query dispatch: `handler.rs:466` (`Message::Query` arm) → `dispatch_query` (handler.rs:262). +- TS client `query(query, opts?)` already takes an options object as arg 2 (`clients/ts/src/index.ts:226`) — the params overload must disambiguate with `Array.isArray`. +- REPL loops: embedded at `crates/cli/src/main.rs:919`, remote at 1092; `--exec` one-shot mode exists (main.rs:520-543) — the eval harness uses it. +- Executor test style: `crates/query/src/executor/tests.rs:10-27` (`test_engine()` helper, temp dir + counter). Btree test style: `btree.rs:1904+` (`temp_btree`). Protocol test style: `protocol.rs:339+`. + +--- + +## Task 1: EXPLAIN shows the lowered plan (unindexed IndexScan → Filter(SeqScan)) + +**Files:** +- Modify: `crates/query/src/executor/plan_exec.rs` (`lower_unindexed_range_scans` at 3372, rename to `lower_unindexed_scans`; doc comment 3360–3371) +- Modify: `crates/query/src/executor/mod.rs` (import at lines 223–224; call sites at 438, 461, 477, 501, 579, 590, 597) +- Test: `crates/query/src/executor/tests.rs` (EXPLAIN section starts at 3254) + +**Steps:** + +- [ ] 1. Write failing tests in `crates/query/src/executor/tests.rs` after `test_explain_filter` (line ~3293), matching the existing helper style: + +```rust +fn explain_text(engine: &mut Engine, q: &str) -> String { + match engine.execute_powql(q).unwrap() { + QueryResult::Rows { rows, .. } => rows + .iter() + .map(|r| match &r[0] { + Value::Str(s) => s.as_str(), + _ => "", + }) + .collect::>() + .join("\n"), + _ => panic!("expected rows"), + } +} + +#[test] +fn test_explain_eq_filter_unindexed_shows_seqscan_not_indexscan() { + let mut engine = test_engine(); + // `email` has NO index in test_engine; the planner folds + // `.email = lit` to IndexScan speculatively. EXPLAIN must show + // what actually runs: Filter over SeqScan. + let text = explain_text(&mut engine, r#"explain User filter .email = "alice@ex.com""#); + assert!(!text.contains("IndexScan"), "got: {text}"); + assert!(text.contains("Filter"), "got: {text}"); + assert!(text.contains("SeqScan"), "got: {text}"); +} + +#[test] +fn test_explain_eq_filter_indexed_shows_indexscan() { + let mut engine = test_engine(); + engine.execute_powql("alter User add index .email").unwrap(); + let text = explain_text(&mut engine, r#"explain User filter .email = "alice@ex.com""#); + assert!(text.contains("IndexScan"), "got: {text}"); +} +``` + +- [ ] 2. Run them and confirm the failure shape: +```bash +cargo test -p powdb-query test_explain_eq_filter -- --nocapture +``` +Expected: `test_explain_eq_filter_unindexed_shows_seqscan_not_indexscan` FAILS (text contains `IndexScan table=User column=email …`); the indexed test passes already. + +- [ ] 3. Implement: in `plan_exec.rs`, rename `lower_unindexed_range_scans` → `lower_unindexed_scans` (it now covers both speculative leaf kinds) and add an `IndexScan` arm before the catch-all, reusing the exact predicate shape the runtime fallback synthesizes at 1620–1624: + +```rust +PlanNode::IndexScan { table, column, key } => { + if let Some(tbl) = catalog.get_table(table) { + if tbl.has_index(column) { + return plan.clone(); + } + } + PlanNode::Filter { + input: Box::new(PlanNode::SeqScan { table: table.clone() }), + predicate: Expr::BinaryOp( + Box::new(Expr::Field(column.clone())), + BinOp::Eq, + Box::new(key.clone()), + ), + } +} +``` +Update the function doc comment (3360–3371) to say it lowers BOTH unindexed `RangeScan` and unindexed `IndexScan`. Update the `use` in `executor/mod.rs:224` and all 7 call sites to the new name. + +- [ ] 4. Behavioral note to verify while implementing (not a TBD — a check): `Update(IndexScan)` plans (planner.rs:447) on unindexed columns now lower to `Update(Filter(SeqScan))`, which hits the fused scan+update path (plan_exec.rs:844–870) instead of `collect_rids_for_mutation`'s IndexScan fallback (plan_exec.rs:2715). Both are correct; run the executor update tests explicitly: `cargo test -p powdb-query -- update`. + +- [ ] 5. Run the full GATE. The existing `test_explain_filter` (tests.rs:3271, expects `Filter` for an unindexed range) must still pass — the RangeScan lowering for `Explain` was already in place. + +- [ ] 6. Commit: +```bash +git add crates/query +git commit -m "fix(query): lower unindexed IndexScan in plan lowering so EXPLAIN shows the executed plan + +EXPLAIN previously printed the planner's speculative IndexScan even when +no index existed and execution fell back to a filtered scan. The lowering +pass (renamed lower_unindexed_scans) now rewrites unindexed IndexScan to +Filter(SeqScan), same as it already did for RangeScan." +``` + +--- + +## Task 2: Range scans use B+tree indexes (non-unique composite-key traversal) + +**Files:** +- Modify: `crates/storage/src/btree.rs` (new `range_rids` method near the non-unique section, after `lookup_prefix_int` at 925; tests in the `Non-unique index tests` section at 1904+) +- Modify: `crates/query/src/executor/plan_exec.rs` (RangeScan exec arm 1659–1699+; lowering gate at 3385; comment 3381–3384) +- Modify: `crates/bench/benches/powql.rs` (new bench fn; register in `criterion_group!` at 655) +- Modify docs: `docs/POWQL.md:1035`, `docs/getting-started.md:368`, `AGENTS.md:130` (perf section item 1) +- Test: `crates/storage/src/btree.rs` tests, `crates/query/src/executor/tests.rs` + +**Steps:** + +- [ ] 1. Write failing btree unit test in `btree.rs` tests (style of `test_non_unique_insert_and_lookup_prefix` at 1907, using `temp_btree`): + +```rust +#[test] +fn test_non_unique_range_rids() { + let mut bt = temp_btree("nonunique_range"); + let rids: Vec = (0..6u32) + .map(|i| RowId { page_id: i, slot_index: 0 }) + .collect(); + for (i, rid) in rids.iter().enumerate() { + bt.insert_non_unique_int((i as i64) * 10, *rid); // 0,10,20,30,40,50 + } + // 10 <= v <= 30 → rids[1..=3] + let hits = bt.range_rids(Some(&Value::Int(10)), Some(&Value::Int(30))); + assert_eq!(hits, vec![rids[1], rids[2], rids[3]]); + // unbounded below + let hits = bt.range_rids(None, Some(&Value::Int(10))); + assert_eq!(hits, vec![rids[0], rids[1]]); + // unbounded above + let hits = bt.range_rids(Some(&Value::Int(40)), None); + assert_eq!(hits, vec![rids[4], rids[5]]); + // duplicates within the range all come back + bt.insert_non_unique_int(20, RowId { page_id: 99, slot_index: 7 }); + let hits = bt.range_rids(Some(&Value::Int(20)), Some(&Value::Int(20))); + assert_eq!(hits.len(), 2); +} +``` + +- [ ] 2. `cargo test -p powdb-storage test_non_unique_range_rids` — expected failure: `no method named range_rids found for struct BTree` (compile error). + +- [ ] 3. Implement `range_rids` in `btree.rs` (after `lookup_prefix_int`, line ~927). Bounds are always INCLUSIVE at the composite level — exclusivity is enforced by the executor's per-row recheck (step 7): + +```rust +/// Range scan over a NON-unique index: return RowIds for all entries +/// whose column value lies in [start, end] (inclusive; pass None for +/// an unbounded side). Composite-key bounds reuse the prefix encoding: +/// (start, RowId::MIN) .. (end, RowId::MAX). Caller rechecks exclusive +/// bounds against the decoded row. +pub fn range_rids(&self, start: Option<&Value>, end: Option<&Value>) -> Vec { + let collect = |pairs: Vec<(Value, RowId)>| { + pairs + .into_iter() + .filter_map(|(k, _)| Self::rid_from_composite(&k)) + .collect() + }; + match (start, end) { + (Some(s), Some(e)) => { + let lo = Self::make_prefix_start(s); + let hi = Self::make_prefix_end(e); + self.range(&lo, &hi) + .filter_map(|(k, _)| Self::rid_from_composite(&k)) + .collect() + } + (Some(s), None) => collect(self.range_from(&Self::make_prefix_start(s))), + (None, Some(e)) => collect(self.range_to(&Self::make_prefix_end(e))), + (None, None) => collect(self.range_from(&Self::make_prefix_start(&Value::Empty))), + } +} +``` +Note on `(None, None)`: a single-column index holds one value type per tree, so full-tree iteration via the leftmost leaf is fine; but the executor never sends `(None, None)` (it short-circuits to a heap scan at plan_exec.rs:1693–1697). Implement the simple correct thing and keep the executor short-circuit. + +- [ ] 4. Run `cargo test -p powdb-storage` — green. + +- [ ] 5. Write failing executor tests in `executor/tests.rs`: + +```rust +#[test] +fn test_range_scan_uses_nonunique_index_same_results() { + let mut engine = test_engine(); // Alice 30, Bob 25, Charlie 35 + let unindexed = engine.execute_powql("User filter .age > 26 and .age <= 35 { .name }").unwrap(); + engine.execute_powql("alter User add index .age").unwrap(); + let indexed = engine.execute_powql("User filter .age > 26 and .age <= 35 { .name }").unwrap(); + let names = |r: QueryResult| match r { + QueryResult::Rows { rows, .. } => { + let mut v: Vec = rows.iter().map(|r| format!("{:?}", r[0])).collect(); + v.sort(); + v + } + _ => panic!("expected rows"), + }; + assert_eq!(names(unindexed), names(indexed)); // Alice, Charlie +} + +#[test] +fn test_range_scan_indexed_excludes_nulls() { + let mut engine = test_engine(); + engine.execute_powql(r#"insert User { name := "Dana", email := "d@ex.com" }"#).unwrap(); // age null + engine.execute_powql("alter User add index .age").unwrap(); + match engine.execute_powql("User filter .age < 100 { .name }").unwrap() { + QueryResult::Rows { rows, .. } => assert_eq!(rows.len(), 3, "null age must not match"), + _ => panic!("expected rows"), + } +} + +#[test] +fn test_explain_range_indexed_shows_rangescan() { + let mut engine = test_engine(); + engine.execute_powql("alter User add index .age").unwrap(); + let text = explain_text(&mut engine, "explain User filter .age > 26"); + assert!(text.contains("RangeScan"), "got: {text}"); +} +``` +Also add a `between` case (`User filter .age between 25 and 30`) and an exclusive-bound case asserting Bob(25) is excluded by `.age > 25`. + +- [ ] 6. Run `cargo test -p powdb-query test_range_scan -- --nocapture` and `test_explain_range_indexed` — expected: `test_explain_range_indexed_shows_rangescan` fails with `Filter`/`SeqScan` text (non-unique indexes get lowered today); the equality-of-results test passes trivially pre-change (both lowered) — it becomes the regression net post-change. + +- [ ] 7. Implement executor support in `plan_exec.rs` RangeScan arm: after the existing unique-index branch (1687), add: + +```rust +// Non-unique index: composite-key leaf walk, then heap fetch + recheck. +// The recheck enforces exclusive bounds (range_rids is inclusive) and +// defensively skips any decoded null. +if tbl.is_index_unique(column) == Some(false) { + if let Some(btree) = tbl.index(column) { + if start_val.is_some() || end_val.is_some() { + let col_idx = schema.column_index(column).ok_or_else(|| { + QueryError::ColumnNotFound { table: String::new(), column: column.clone() } + })?; + let rids = btree.range_rids(start_val.as_ref(), end_val.as_ref()); + let mut rows: Vec> = Vec::with_capacity(rids.len()); + for rid in rids { + if let Some(data) = tbl.heap.get(rid) { + let row = decode_row(&tbl.schema, &data); + if !row[col_idx].is_empty() + && range_matches(&row[col_idx], &start_val, start_inclusive, &end_val, end_inclusive) + { + rows.push(row); + } + } + } + return Ok(QueryResult::Rows { columns, rows }); + } + } +} +``` +Then flip the lowering gate at 3385 from `tbl.is_index_unique(column) == Some(true)` to `tbl.has_index(column)`, and rewrite the comment at 3381–3384 (non-unique indexes now traverse composite keys natively). + +- [ ] 8. Run `cargo test -p powdb-query` — all range/explain tests green, including Task 1's tests and the pre-existing `test_explain_filter` (unindexed range still lowers). + +- [ ] 9. Bench: add to `crates/bench/benches/powql.rs` (mirror `bench_powql_filter_only` at 218 — copy its setup, add `alter ... add index .age` during setup) a `bench_range_scan_indexed` and register it in `criterion_group!` (line 655). Informal check only: +```bash +cargo bench -p powdb-bench -- range_scan_indexed filter_only point_lookup +``` +Decision criteria: indexed range scan on a selective range should beat `filter_only` for selectivity well under ~10%; `filter_only` and `point_lookup` numbers must be within noise of the pre-change run (re-run the pre-change baseline once before starting the task and keep the console output). Do NOT modify `crates/bench/baseline/main.json`. + +- [ ] 10. Docs in the SAME commit: update `docs/POWQL.md:1035` and `docs/getting-started.md:368` (the docs sweep just corrected these to say range scans do NOT use indexes — flip them back once true); update `AGENTS.md:130` item 1 to: planner emits `RangeScan`/`IndexScan` speculatively; executor lowers to `Filter(SeqScan)` only when no index exists, otherwise walks the B+tree (unique: raw keys; non-unique: composite `(value, rid)` keys). + +- [ ] 11. Full GATE, then commit: +```bash +git add crates/storage crates/query crates/bench docs AGENTS.md +git commit -m "feat(query,storage): RangeScan executes against non-unique B+tree indexes + +BTree::range_rids walks composite (value,rid) keys between prefix bounds; +the executor fetches rows from the heap by rid and rechecks bounds (which +also preserves null-exclusion semantics). Plan lowering now keeps RangeScan +whenever ANY index exists on the column. Adds range_scan_indexed criterion +workload; docs updated to match." +``` + +--- + +## Task 3: UNIQUE constraints (`unique` field modifier, enforced insert/update/upsert, `alter T add unique .col`) + +Design (decided): `unique` modifier only — no `primary` keyword this sprint. Uniqueness lives where it already half-exists: `IndexedCol.unique` / persisted `IndexedColMeta`. Declaring `unique` auto-creates a unique B+tree index. Enforcement is in the storage layer (`Table::insert` / update paths) so every write path — plain, prepared (`insert_by_slot`), upsert — passes one choke point. `upsert ... on .col` now REQUIRES `.col` to be unique (breaking change; fixes the known duplicate-id bug). No new `ColumnDef` flag needed. + +**Files:** +- Modify: `crates/query/src/token.rs` (new `Token::Unique`; `display_name` at 253-area), `crates/query/src/lexer.rs` (keyword map at ~263), `crates/query/src/canonicalize.rs` (token hash match at ~250–275 — exhaustive, compiler will force the arm; pick the next unused hash byte after auditing `0x7B+`) +- Modify: `crates/query/src/ast.rs` (`FieldDef` gets `unique: bool` at 195–198; `AlterAction::AddUnique { column }` at 31) +- Modify: `crates/query/src/parser.rs` (`parse_create_type` modifier loop at 1771–1777; `parse_alter_table` at 1544–1563; `tokens_to_text` at 1849-area) +- Modify: `crates/query/src/plan.rs` (`CreateTable.fields: Vec<(String, String, bool)>` at 116–119 → `Vec` struct with `name/type_name/required/unique`) +- Modify: `crates/query/src/planner.rs` (CreateType arm), `crates/query/src/executor/plan_exec.rs` (CreateTable arm at 1393; Upsert arm at 695; AlterTable arm at 1418) +- Modify: `crates/storage/src/table.rs` (`Table::insert` at 374 — pre-check BEFORE heap insert; `update`/`update_hinted` at 799/813) +- Test: `crates/query/src/parser.rs` tests, `crates/query/src/executor/tests.rs`, `crates/storage/src/table.rs`/`catalog.rs` tests +- Docs (same commit): `docs/POWQL.md` (type DDL + alter + cheat sheet), `AGENTS.md` (cheat-sheet row + footguns), `site/powql.html` + +**Steps:** + +- [ ] 1. **Investigation (numbered, explicit):** + 1. Confirm `IndexedColMeta.unique` round-trips through `Catalog::persist()`/`Catalog::open` (read the catalog.bin serializer near catalog.rs:1816 where index-list back-compat is handled). Decision: if `unique` is already serialized, nothing to do; if it is serialized as name-only, extend the record with a trailing unique byte using the same append-only back-compat pattern as the Connect message. + 2. Enumerate every write path that can change an indexed column's value and confirm each flows through `Table::insert`, `Table::update`, or `Table::update_hinted`: the byte-patch fast paths are excluded by construction (`no_indexed` guard at plan_exec.rs:891–893; `has_indexed_col` guard at prepared.rs:225); check `scan_patch_matching_with_hook` (table.rs:767) — if its hook updates indexes, add the same guard there. Decision criteria: any path that can write a duplicate into a unique btree must either be guarded or be unreachable for indexed columns; document the audit in the commit message. + 3. Confirm there is no `drop index` statement (grep `DropIndex` in `crates/query/src`). Consequence: `alter T add unique .col` on a column that already has a NON-unique index must be a clean error ("column already indexed"), not an in-place upgrade. + +- [ ] 2. Failing parser tests (style of `test_parse_alter_add_required_column` at parser.rs:2870): + +```rust +#[test] +fn test_parse_type_with_unique_modifier() { + let stmt = parse("type User { required unique email: str, age: int }").unwrap(); + match stmt { + Statement::CreateType(ct) => { + assert!(ct.fields[0].required && ct.fields[0].unique); + assert!(!ct.fields[1].unique); + } + other => panic!("expected CreateType, got {other:?}"), + } +} + +#[test] +fn test_parse_alter_add_unique() { + let stmt = parse("alter User add unique .email").unwrap(); + match stmt { + Statement::AlterTable(at) => assert!(matches!( + at.action, + AlterAction::AddUnique { ref column } if column == "email" + )), + other => panic!("expected AlterTable, got {other:?}"), + } +} +``` +Run: `cargo test -p powdb-query test_parse_type_with_unique` → compile error (`no field unique on FieldDef`) — that is the expected failure. + +- [ ] 3. Implement lexer/token/AST/parser: + - `token.rs`: `Unique, // unique` next to `Index` (line 26); `Token::Unique => "'unique'".into()` in `display_name`. + - `lexer.rs` keyword map (~263): `"unique" => Token::Unique,`. + - `canonicalize.rs`: add `Token::Unique => hash_byte(h, )` (audit the match for the first unused value > 0x7B). + - `parser.rs::parse_create_type`: replace the single `required` check (1772–1777) with a small modifier loop accepting `required` and `unique` in either order: + ```rust + let (mut required, mut unique) = (false, false); + loop { + match self.peek() { + Token::Required => { self.advance(); required = true; } + Token::Unique => { self.advance(); unique = true; } + _ => break, + } + } + ``` + - `parse_alter_table`: after `Token::Add` (1545), before the `index` check, handle `Token::Unique` → expect `DotIdent` → `AlterAction::AddUnique { column }`. + - `tokens_to_text` (1849-area): `Token::Unique => out.push_str("unique")`. + - `ast.rs`: `FieldDef { name, type_name, required, unique }`; `AlterAction::AddUnique { column: String }`. Fix all construction sites the compiler flags (parser.rs:1797–1801 and tests). + +- [ ] 4. Plan + planner + executor DDL: change `PlanNode::CreateTable.fields` to a named struct in `plan.rs`: +```rust +#[derive(Debug, Clone)] +pub struct CreateField { + pub name: String, + pub type_name: String, + pub required: bool, + pub unique: bool, +} +``` +Update the planner's CreateType arm and the executor's CreateTable arm (plan_exec.rs:1393): after `catalog.create_table(schema)`, loop unique fields → `self.catalog.create_index_unique(name, &f.name, true)`. Add the `AlterAction::AddUnique` executor arm (next to AddIndex at 1451): scan the table, collect values into a `std::collections::HashSet`, skip `Value::Empty`; on first duplicate return `Err(QueryError::Execution(format!("cannot add unique on {table}.{column}: duplicate value {v:?} exists")))`; if `tbl.has_index(column)` already → `Err(... "column already indexed")`; else `catalog.create_index_unique(table, column, true)`. (`plan_cache.rs` CreateTable arms at 234/365/711 are `{ .. }` wildcards — no change.) + +- [ ] 5. Failing enforcement tests in `executor/tests.rs` (write all five BEFORE the storage change): + +```rust +fn unique_engine() -> Engine { + let id = TEST_COUNTER.fetch_add(1, Ordering::SeqCst); + let dir = std::env::temp_dir().join(format!("powdb_uniq_{}_{}", std::process::id(), id)); + let mut engine = Engine::new(&dir).unwrap(); + engine.execute_powql("type Acct { required unique email: str, id: int }").unwrap(); + engine.execute_powql(r#"insert Acct { email := "a@x.com", id := 1 }"#).unwrap(); + engine +} + +#[test] +fn test_unique_dup_insert_rejected() { + let mut engine = unique_engine(); + let err = engine + .execute_powql(r#"insert Acct { email := "a@x.com", id := 2 }"#) + .unwrap_err(); + assert!(err.to_string().contains("unique constraint violation on Acct.email"), "{err}"); + match engine.execute_powql("count(Acct)").unwrap() { + QueryResult::Scalar(Value::Int(n)) => assert_eq!(n, 1), + other => panic!("expected scalar, got {other:?}"), + } +} + +#[test] +fn test_unique_update_into_dup_rejected() { + let mut engine = unique_engine(); + engine.execute_powql(r#"insert Acct { email := "b@x.com", id := 2 }"#).unwrap(); + let err = engine + .execute_powql(r#"Acct filter .id = 2 update { email := "a@x.com" }"#) + .unwrap_err(); + assert!(err.to_string().contains("unique constraint violation"), "{err}"); +} + +#[test] +fn test_upsert_requires_unique_and_no_dup_ids() { + let id = TEST_COUNTER.fetch_add(1, Ordering::SeqCst); + let dir = std::env::temp_dir().join(format!("powdb_ups_{}_{}", std::process::id(), id)); + let mut engine = Engine::new(&dir).unwrap(); + engine.execute_powql("type W { unique id: int, v: str }").unwrap(); + engine.execute_powql(r#"upsert W on .id { id := 1, v := "first" }"#).unwrap(); + // Known bug regression: a plain insert of the same id must now fail + // instead of silently creating a second id=1 row. + assert!(engine.execute_powql(r#"insert W { id := 1, v := "second" }"#).is_err()); + engine.execute_powql(r#"upsert W on .id { id := 1, v := "third" }"#).unwrap(); + match engine.execute_powql("count(W)").unwrap() { + QueryResult::Scalar(Value::Int(n)) => assert_eq!(n, 1), + other => panic!("expected scalar, got {other:?}"), + } + // upsert on a NON-unique column is a clean error. + engine.execute_powql("type W2 { id: int }").unwrap(); + let err = engine.execute_powql("upsert W2 on .id { id := 1 }").unwrap_err(); + assert!(err.to_string().contains("requires a unique column"), "{err}"); +} + +#[test] +fn test_alter_add_unique_fails_on_existing_dups() { + let id = TEST_COUNTER.fetch_add(1, Ordering::SeqCst); + let dir = std::env::temp_dir().join(format!("powdb_audup_{}_{}", std::process::id(), id)); + let mut engine = Engine::new(&dir).unwrap(); + engine.execute_powql("type L { e: str }").unwrap(); + engine.execute_powql(r#"insert L { e := "x" }"#).unwrap(); + engine.execute_powql(r#"insert L { e := "x" }"#).unwrap(); + assert!(engine.execute_powql("alter L add unique .e").is_err()); +} + +#[test] +fn test_unique_constraint_survives_reopen() { + let id = TEST_COUNTER.fetch_add(1, Ordering::SeqCst); + let dir = std::env::temp_dir().join(format!("powdb_uniq_re_{}_{}", std::process::id(), id)); + { + let mut engine = Engine::new(&dir).unwrap(); + engine.execute_powql("type Acct { required unique email: str }").unwrap(); + engine.execute_powql(r#"insert Acct { email := "a@x.com" }"#).unwrap(); + // Dropped here without explicit checkpoint — recovery path must + // restore the unique flag from catalog.bin + WAL replay. + } + let mut engine = Engine::new(&dir).unwrap(); + assert!(engine.execute_powql(r#"insert Acct { email := "a@x.com" }"#).is_err()); +} +``` + +- [ ] 6. Run: `cargo test -p powdb-query test_unique -- --nocapture` and `test_upsert_requires` — expected failures: dup insert currently succeeds (count = 2), upsert on non-unique col currently succeeds, alter accepts dups (no AddUnique arm yet → compile error first; fix arms in step 4 before this run). + +- [ ] 7. Implement storage enforcement in `table.rs`: + - In `Table::insert` (374): BEFORE `self.heap.insert(...)` (line 381), pre-check unique columns: + ```rust + for entry in &self.indexed_cols { + if !entry.unique { continue; } + let val = &values[entry.col_idx]; + if !val.is_empty() && entry.btree.lookup(val).is_some() { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("unique constraint violation on {}.{}", + self.schema.table_name, entry.col_name), + )); + } + } + ``` + - In `Table::update` / `update_hinted` (799/813): where the old row's index entries are diffed against new values, add the same check but allow `existing_rid == rid` (updating a row to its own current value is legal). Apply the audit findings from step 1.2 to `scan_patch_matching_with_hook` if needed. + - Verify the error surfaces usefully: the executor wraps as `QueryError::StorageError(e.to_string())` — check `QueryError`'s `Display` in `result.rs`; the assertion is `contains("unique constraint violation on …")`, so a prefix is acceptable. + +- [ ] 8. Implement Upsert gate in plan_exec.rs:695: at the top of the arm (after schema resolution), require `tbl.is_index_unique(key_column) == Some(true)`, else `Err(QueryError::Execution(format!("upsert on .{key_column} requires a unique column (declare it with `unique {key_column}: ` or `alter {table} add unique .{key_column}`)")))`. The probe at 739–749 then always uses the unique-btree `index_lookup_all` path; the no-index linear scan branch (750–760) becomes unreachable — delete it. + +- [ ] 9. Run all of step 5's tests plus the full `cargo test -p powdb-query` and `cargo test -p powdb-storage`. Existing tests that upsert on non-unique columns (grep `upsert` in `executor/tests.rs` and `clients/ts/test/client.test.ts`) must be updated to declare the key column `unique` — that's part of the breaking change, do it deliberately, and update `clients/ts` tests in this commit too if they exercise upsert against the dev server. + +- [ ] 10. Docs in the same commit: `docs/POWQL.md` type/alter sections + cheat-sheet row (`unique email: str` / `CREATE TABLE ... UNIQUE`); `AGENTS.md` cheat-sheet row + a footgun note ("upsert requires the `on` column to be unique since 0.4.7"); `site/powql.html` matching row. + +- [ ] 11. Full GATE; commit: +```bash +git add crates docs site AGENTS.md clients/ts +git commit -m "feat(powql): unique constraints — unique field modifier, alter add unique, enforced on insert/update/upsert + +Declaring unique auto-creates a unique B+tree index; enforcement is a +storage-layer pre-check before any heap write, so plain, prepared, and +upsert paths share one choke point. upsert on .col now requires .col to +be unique (fixes upsert-then-insert duplicate-id bug). alter T add unique +.col scans for existing duplicates first. Constraint survives restart via +persisted IndexedColMeta + WAL replay index rebuild." +``` + +--- + +## Task 4: Parameter binding over the wire (`$1..$N` placeholders) + +Design (decided): placeholders are `$1`-style (1-based; `?` is unusable because `??` is the COALESCE token, lexer.rs:302). Binding happens at TOKEN level inside the query crate: lex the template, replace each `Token::Param(n)` with the literal token for `params[n-1]`, then parse normally. Values are never re-lexed or string-interpolated — a string param becomes a `Token::StrLit` payload byte-for-byte, so injection shapes are inert data. This also sidesteps `Literal` having no Null variant (`Token::Null` substitutes directly). v1 bypasses the plan cache (template caching is a follow-up). + +**Files:** +- Modify: `crates/query/src/token.rs` (`Token::Param(u32)`), `crates/query/src/lexer.rs` (`$` + digits), `crates/query/src/canonicalize.rs` (new arm), `crates/query/src/parser.rs` (`pub fn parse_with_params`), `crates/query/src/ast.rs` or `result.rs` (`pub enum ParamValue { Null, Int(i64), Float(f64), Bool(bool), Str(String) }`), `crates/query/src/executor/mod.rs` (`execute_powql_with_params` + `execute_powql_readonly_with_params`, mirroring the non-cached paths at 475–487 and 556–598) +- Modify: `crates/server/src/protocol.rs` (`MSG_QUERY_PARAMS = 0x04`, `Message::QueryWithParams`), `crates/server/src/handler.rs` (new match arm next to `Message::Query` at 466; `dispatch_query_with_params` beside `dispatch_query` at 262 — parse via `parse_with_params` for role enforcement, then route read/write exactly like the existing fn) +- Modify: `clients/ts/src/protocol.ts` (MSG_QUERY_PARAMS + encode), `clients/ts/src/index.ts` (`query(query, params?, opts?)`), `clients/ts/README.md` +- Test: `crates/query/src/executor/tests.rs`, `crates/server/src/protocol.rs` tests, `clients/ts/test/client.test.ts`, `clients/ts/test/protocol.test.ts` +- Docs: `AGENTS.md:186` (replace the escape-it-yourself paragraph), `docs/POWQL.md` (placeholders note) + +**Steps:** + +- [ ] 1. Failing engine-level test in `executor/tests.rs`: + +```rust +#[test] +fn test_params_bind_injection_shaped_strings_byte_faithfully() { + let mut engine = test_engine(); + let evil = r#"x"; drop User; filter .age > "0"#; + engine + .execute_powql_with_params( + "insert User { name := $1, email := $2, age := $3 }", + &[ + ParamValue::Str(evil.to_string()), + ParamValue::Str("e@x.com".into()), + ParamValue::Int(40), + ], + ) + .unwrap(); + let r = engine + .execute_powql_with_params( + "User filter .email = $1 { .name }", + &[ParamValue::Str("e@x.com".into())], + ) + .unwrap(); + match r { + QueryResult::Rows { rows, .. } => { + assert_eq!(rows.len(), 1); + assert_eq!(rows[0][0], Value::Str(evil.to_string())); + } + other => panic!("expected rows, got {other:?}"), + } + // Table survived; 4 rows total. + match engine.execute_powql("count(User)").unwrap() { + QueryResult::Scalar(Value::Int(n)) => assert_eq!(n, 4), + other => panic!("{other:?}"), + } +} + +#[test] +fn test_params_errors() { + let mut engine = test_engine(); + // Out-of-range and unbound placeholders are clean errors. + assert!(engine.execute_powql_with_params("User filter .age > $2", &[ParamValue::Int(1)]).is_err()); + assert!(engine.execute_powql("User filter .age > $1").is_err()); // no-params API + // Null param round-trips as PowQL null. + engine.execute_powql_with_params( + "insert User { name := $1, email := $2, age := $3 }", + &[ParamValue::Str("N".into()), ParamValue::Str("n@x.com".into()), ParamValue::Null], + ).unwrap(); + match engine.execute_powql("User filter .age = null { .name }").unwrap() { + QueryResult::Rows { rows, .. } => assert_eq!(rows.len(), 1), + other => panic!("{other:?}"), + } +} +``` +Run `cargo test -p powdb-query test_params` → compile errors (no `ParamValue`, no method) — expected. + +- [ ] 2. Implement the query-crate half: + - `token.rs`: `Param(u32), // $1` + `display_name` → `format!("parameter ${n}")`. + - `lexer.rs`: on `'$'`, consume digits; empty digits or value 0 → lex error `"expected parameter number after '$'"`; emit `Token::Param(n)`. + - `canonicalize.rs`: `Token::Param(n) => { hash_byte(h, ); hash for n bytes }` — match how `IntLit` is hashed in that file (read the IntLit arm and mirror its structure). + - `parser.rs`: + ```rust + pub fn parse_with_params(input: &str, params: &[ParamValue]) -> Result { + let mut tokens = lex(input).map_err(|e| ParseError::Lex { message: e.message, position: e.position })?; + for tok in tokens.iter_mut() { + if let Token::Param(n) = tok { + let idx = (*n as usize) - 1; + let p = params.get(idx).ok_or_else(|| ParseError::Syntax { + message: format!("query references ${n} but only {} parameter(s) were supplied", params.len()), + })?; + *tok = match p { + ParamValue::Null => Token::Null, + ParamValue::Int(v) => Token::IntLit(*v), + ParamValue::Float(v) => Token::FloatLit(*v), + ParamValue::Bool(v) => Token::BoolLit(*v), + ParamValue::Str(s) => Token::StrLit(s.clone()), + }; + } + } + // …construct Parser { tokens, pos: 0, depth: 0 }, parse_statement, trailing-token check — + // factor the shared tail of `parse` (parser.rs:99-124) into a helper used by both. + } + ``` + (Verify the exact string-token variant name — `tokens_to_text` at parser.rs:1814+ shows the canonical spellings; use what `token.rs:6-12` declares.) + In plain `parse`, leave `Token::Param` unhandled by expression parsing so it produces the existing "unexpected token" error path; the new `display_name` makes the error self-explanatory. + - `executor/mod.rs`: `pub fn execute_powql_with_params(&mut self, input: &str, params: &[ParamValue])` = `parse_with_params` → `planner::plan_statement` → `lower_unindexed_scans` → `execute_plan` → `sync_wal` if `!self.in_transaction` (clone of the lex-error fallback path at 475–487). `pub fn execute_powql_readonly_with_params(&self, ...)` mirrors 556–598: parse, `is_read_only_statement` check → `ReadonlyNeedsWrite`, plan, lower, `execute_plan_readonly`. No plan-cache interaction. + +- [ ] 3. Run step 1 tests → green. `cargo test -p powdb-query` → green (canonicalize/lexer tests unaffected). + +- [ ] 4. Failing protocol round-trip test in `protocol.rs` tests: + +```rust +#[test] +fn test_encode_decode_query_with_params() { + let msg = Message::QueryWithParams { + query: "insert User { name := $1, age := $2, ok := $3, note := $4 }".into(), + params: vec![ + WireParam::Str(r#"a"b\c; drop User"#.into()), + WireParam::Int(-7), + WireParam::Bool(true), + WireParam::Null, + ], + }; + let bytes = msg.encode(); + match Message::decode(&bytes).unwrap() { + Message::QueryWithParams { query, params } => { + assert!(query.contains("$1")); + assert_eq!(params.len(), 4); + assert!(matches!(¶ms[0], WireParam::Str(s) if s == r#"a"b\c; drop User"#)); + } + other => panic!("expected QueryWithParams, got {other:?}"), + } +} +``` +Plus extend `test_decode_garbage_never_panics` with truncated `0x04` frames (bad tag byte, truncated i64, truncated string). + +- [ ] 5. Implement protocol + handler: + - `protocol.rs`: `const MSG_QUERY_PARAMS: u8 = 0x04;`; `pub enum WireParam { Null, Int(i64), Float(f64), Bool(bool), Str(String) }`; encode = query string, `u16` count LE, then per param `tag u8` (0 null, 1 int + 8B LE, 2 float + 8B LE, 3 bool + 1B, 4 str + length-prefixed). Decode strictly, unknown tag → `Err("unknown param tag")`. Version-gating: this is a NEW message type — old clients never send it (unchanged frames), old servers answer it with the existing `unknown message type: 0x4` error; no existing message changes shape. + - `handler.rs`: `Message::QueryWithParams { query, params }` arm cloned from the `Query` arm at 466–501 (same `MAX_QUERY_LENGTH` check, same `spawn_blocking` + timeout), calling a new `dispatch_query_with_params(&engine, &query, ¶ms, principal)` that converts `WireParam` → `ParamValue`, parses once via `parse_with_params` for `check_statement_permitted` + `is_read_only_statement`, then `execute_powql_readonly_with_params` under `.read()` with `ReadonlyNeedsWrite` escalation to `.write()` + `execute_powql_with_params` — structurally identical to `dispatch_query` (262–297). + +- [ ] 6. `cargo test -p powdb-server` → green; full GATE; commit the Rust half: +```bash +git add crates/query crates/server docs/POWQL.md AGENTS.md +git commit -m "feat(query,server): parameter binding — \$N placeholders bound at token level, QueryWithParams wire message + +Params are substituted as literal tokens before parsing (never re-lexed, +never string-interpolated), so untrusted input cannot change query shape. +New MSG_QUERY_PARAMS (0x04) is a pure protocol addition; existing +messages and old clients are untouched." +``` +(AGENTS.md edit: replace the "**No parameter binding yet.** … escape it yourself" paragraph with a `$1` usage example; POWQL.md gets a placeholders subsection.) + +- [ ] 7. TS client: failing tests first — in `clients/ts/test/client.test.ts` (existing homegrown `test()` harness): +```ts +await test("query with params stores injection-shaped strings byte-faithfully", async () => { + await client.query(`type ${tbl("P")} { required name: str, age: int }`); + const evil = `x"; drop ${tbl("P")}; filter .age > "0`; + const ins = await client.query(`insert ${tbl("P")} { name := $1, age := $2 }`, [evil, 9]); + assert.equal(ins.kind, "ok"); + const r = await client.query(`${tbl("P")} filter .age = $1 { .name }`, [9]); + assert.equal(r.kind, "rows"); + if (r.kind === "rows") assert.deepEqual(r.rows, [[evil]]); +}); +await test("old no-params query path still works", async () => { + const r = await client.query(`${tbl("P")} { .name }`); + assert.equal(r.kind, "rows"); +}); +``` +Plus a `protocol.test.ts` encode/decode round-trip for the new frame (existing `tryDecode` style). Implement: `protocol.ts` add `MSG_QUERY_PARAMS = 0x04`, a `QueryWithParams` message variant and encode branch; `index.ts`: +```ts +export type QueryParam = string | number | boolean | null; +async query(query: string, paramsOrOpts?: QueryParam[] | { signal?: AbortSignal }, maybeOpts?: { signal?: AbortSignal }): Promise +``` +with `Array.isArray(paramsOrOpts)` disambiguation (back-compat for the existing 2-arg opts form); numbers encode as int when `Number.isInteger`, float otherwise. Run: +```bash +cd clients/ts && pnpm build && pnpm test && pnpm test:protocol +``` +Update `clients/ts/README.md` (params example + "requires powdb-server >= 0.4.7 for the params form") in the same commit. Commit: +```bash +git add clients/ts +git commit -m "feat(ts-client): client.query(powql, params) using QueryWithParams wire message" +``` + +--- + +## Task 5: Multi-line REPL input + +**Files:** +- Modify: `crates/cli/src/main.rs` (embedded loop at 919–933; remote loop at 1092+; new `needs_continuation` helper + `#[cfg(test)]` module) +- Modify docs: `AGENTS.md:161` ("The REPL is line-oriented…"), `docs/getting-started.md` if it repeats the claim + +**Steps:** + +- [ ] 1. **Investigation:** confirm there is no CLI test harness (`ls crates/cli` shows only `src/main.rs`; check `Cargo.toml` for `[[test]]`). Decision: unit-test the helper inside `main.rs`; no integration harness this sprint. Also read the lexer's string rules (`crates/query/src/lexer.rs`, string branch) to match escape handling exactly — decision criteria: if the lexer supports `\"` escapes inside strings, `needs_continuation` must skip a quote preceded by a backslash; if it does not, a bare `"` always toggles. + +- [ ] 2. Failing unit tests at the bottom of `main.rs`: + +```rust +#[cfg(test)] +mod tests { + use super::needs_continuation; + + #[test] + fn continuation_tracking() { + assert!(needs_continuation("type User {")); + assert!(needs_continuation("type User {\n required name: str,")); + assert!(!needs_continuation("type User { required name: str }")); + // Brace inside a string literal must not count. + assert!(!needs_continuation(r#"insert U { s := "}" }"#)); + assert!(needs_continuation(r#"insert U { s := "}" "#)); + // Parens. + assert!(needs_continuation("count(User filter (")); + assert!(!needs_continuation("count(User)")); + // Nested. + assert!(needs_continuation("insert U { a := (1 + ")); + // Over-closed input is NOT a continuation — let the parser error. + assert!(!needs_continuation("User }")); + } +} +``` +Run: `cargo test -p powdb-cli` → compile error (no `needs_continuation`) — expected. + +- [ ] 3. Implement: + +```rust +/// True when `buffer` has unbalanced `{`/`(` outside string literals, +/// i.e. the REPL should read another line before executing. +fn needs_continuation(buffer: &str) -> bool { + let mut depth: i64 = 0; + let mut in_str = false; + let mut chars = buffer.chars().peekable(); + while let Some(c) = chars.next() { + match c { + '"' if in_str => in_str = false, + '"' => in_str = true, + '\\' if in_str => { chars.next(); } // adjust per step-1 lexer findings + '{' | '(' if !in_str => depth += 1, + '}' | ')' if !in_str => depth -= 1, + _ => {} + } + } + depth > 0 && !in_str +} +``` +Wire into `run_embedded` (919) and `run_remote` (1092): keep a `String` buffer; prompt `"powql> "` when empty, `" ...> "` otherwise; on each line, append + `\n`, and only execute when `!needs_continuation(&buffer)`; meta-commands (`.help` etc., 938) only recognized when the buffer is empty; `ReadlineError::Interrupted` clears the buffer and continues; add the full multi-line statement (joined) to history, not the fragments. + +- [ ] 4. Manual verification (real terminal): paste the multi-line `type` example from `docs/POWQL.md` and a multi-line `insert` into `cargo run -p powdb-cli` and confirm execution on the closing brace. + +- [ ] 5. Update `AGENTS.md:161` to: "The REPL buffers lines until braces/parens balance — multi-line `type`/`insert` paste works; a statement still cannot span two separately-submitted balanced lines." Same-commit doc rule applies. + +- [ ] 6. Full GATE; commit: +```bash +git add crates/cli AGENTS.md docs +git commit -m "feat(cli): multi-line REPL input — buffer until braces/parens balance outside string literals" +``` + +--- + +## Task 6: Agent-DX falsification eval (10-table schema) + +Scaffolding + docs only; no model calls anywhere in CI. Runner is Python 3 stdlib (no new repo deps; jq not guaranteed on runners). + +**Files:** +- Create: `scripts/agent-eval/README.md` (how to run with ANY model given only AGENTS.md; baseline procedure vs SQLite) +- Create: `scripts/agent-eval/schema.powql` (10 tables: users, orders, order_items, products, categories, reviews, addresses, payments, sessions, inventory — using `type`, `required`, `unique` where natural, e.g. `unique email` on users after Task 3) +- Create: `scripts/agent-eval/seed.powql` (deterministic seed rows, enough for non-trivial group/join answers) +- Create: `scripts/agent-eval/sqlite-baseline/schema.sql` + `seed.sql` (same data, for the comparison number) +- Create: `scripts/agent-eval/tasks.json` (~25 tasks) +- Create: `scripts/agent-eval/setup.sh` (builds CLI, creates pristine seeded data dir) +- Create: `scripts/agent-eval/run.py` (scores a candidates JSONL offline) + +**Steps:** + +- [ ] 1. `setup.sh`: `cargo build --release -p powdb-cli`, then create `scripts/agent-eval/.golden-data/` by streaming `schema.powql` + `seed.powql` one statement per line through `target/release/powdb-cli --data-dir scripts/agent-eval/.golden-data --exec ""` (the `--exec` one-shot path exists, main.rs:541-543; one process per statement is fine at seed scale — if startup cost bites, batch via the REPL stdin once Task 5 lands). `.golden-data/` is gitignored (add to `.gitignore`). + +- [ ] 2. `tasks.json` schema (each entry): +```json +{ + "id": "agg-03", + "prompt": "How many orders does each city have? Return city and order count, only cities with at least 2 orders.", + "tables_hint": ["users", "orders", "addresses"], + "check": { "type": "rowcount", "expected": 3 } +} +``` +`check.type` ∈ `rowcount` | `scalar` (exact string compare of the single value) | `rows` (sorted exact match, small results only) | `error` (statement must be rejected — e.g. the unique-violation and `count:`-alias-fails tasks). ~25 tasks covering the gotcha list from AGENTS.md: `:=` vs `=`, `type` not `create table`, aliases (`n: count(.name)` — plus one `error` task asserting `count: count(.name)` fails), group+having, inner/left join with table-order note, IN-subquery, upsert (requires unique after Task 3), null checks (`= null`), between, distinct, transactions (begin/insert/rollback then count), `alter add column` / `add index` / `add unique`, order+limit+offset, case expression, parameterless count(*). + +- [ ] 3. `run.py` (stdlib only): for each line of `candidates.jsonl` (`{"task_id": ..., "statement": ...}`), `shutil.copytree(.golden-data, tmpdir)`, `subprocess.run([cli, "--data-dir", tmpdir, "--exec", stmt], capture_output=True, timeout=30)`, parse stdout (the CLI's table/scalar/affected output — implement a tolerant extractor: scalar = last numeric token of last line; rowcount = count of data lines; verify the exact print format from `print_local_result` in main.rs while implementing), score against `check`, emit `results.json` + a pass-rate summary line per category. Exit code 0 always (scoring tool, not a gate). + +- [ ] 4. `README.md` — the harness contract: "Give the model ONLY: AGENTS.md, `schema.powql`, and one task prompt. The model returns exactly one PowQL statement. Append to `candidates.jsonl`. Run `python3 scripts/agent-eval/run.py candidates.jsonl`. Scoring is offline and model-agnostic." Baseline procedure: same prompts against `sqlite-baseline/schema.sql` with the same model, scored with `sqlite3` and the same check semantics; report the two pass rates side by side. Note explicitly: not wired into CI. + +- [ ] 5. Smoke-test the harness end to end by hand-writing a known-good `candidates.jsonl` (save as `scripts/agent-eval/examples/golden-candidates.jsonl`) for 3 tasks (copy correct statements from POWQL.md) and confirming `run.py` scores 3/3, plus one deliberately wrong statement scoring 0/1. + +- [ ] 6. Update `scripts/README.md` with an `agent-eval/` section. Full GATE (workspace untouched, but run it anyway — the gate is per-task policy); commit: +```bash +git add scripts/agent-eval scripts/README.md .gitignore +git commit -m "feat(scripts): agent-DX falsification eval — 10-table schema, 25 scored tasks, model-agnostic offline runner" +``` + +--- + +## Task 7: Integration pass + +**Files:** +- Modify: `CHANGELOG.md` (`[Unreleased]` section) + +**Steps:** + +- [ ] 1. Full workspace gate from a clean state: +```bash +cargo clean -p powdb-query -p powdb-storage -p powdb-server -p powdb-cli +cargo test --workspace +cargo clippy --workspace --all-targets -- -D warnings +cargo fmt --all -- --check +``` +- [ ] 2. TS client gate: `cd clients/ts && pnpm build && pnpm test && pnpm test:protocol && pnpm test:pool`. +- [ ] 3. Sanity perf run (informal, no baselines touched): `cargo run --release -p powdb-compare` — confirm no workload regressed by an order of magnitude vs a pre-sprint run; `cargo bench -p powdb-bench` console-only. +- [ ] 4. Eval harness smoke: `scripts/agent-eval/setup.sh && python3 scripts/agent-eval/run.py scripts/agent-eval/examples/golden-candidates.jsonl`. +- [ ] 5. Update `CHANGELOG.md` `[Unreleased]`: `### Added` — EXPLAIN shows executed (lowered) plan; B+tree range scans on all indexed columns; `unique` constraints (`unique` modifier, `alter T add unique .col`); wire parameter binding (`$N`, new `QueryWithParams` message, TS `query(q, params)`); multi-line REPL input; agent-DX eval harness. `### Changed` (breaking) — `upsert ... on .col` now requires `.col` to be unique. +- [ ] 6. Commit: `git add CHANGELOG.md && git commit -m "chore: easy-wins sprint integration pass — changelog for all six features"`. +- [ ] 7. Do NOT merge or push to `main`; leave `feat/easy-wins-sprint` for review (`superpowers:finishing-a-development-branch`). + +--- + +### Critical Files for Implementation +- `crates/query/src/executor/plan_exec.rs` +- `crates/storage/src/table.rs` +- `crates/storage/src/btree.rs` +- `crates/query/src/parser.rs` +- `crates/server/src/protocol.rs` diff --git a/docs/superpowers/plans/2026-06-09-week1-v046-security-release.md b/docs/superpowers/plans/2026-06-09-week1-v046-security-release.md new file mode 100644 index 0000000..8d57937 --- /dev/null +++ b/docs/superpowers/plans/2026-06-09-week1-v046-security-release.md @@ -0,0 +1,97 @@ +# Week 1 — v0.4.6 Security Release + TS Client Multi-User Auth + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship v0.4.6 (oversized-row DoS fix + readonly enforcement + NULL wire fix + window-frame fix) to crates.io/ghcr/GitHub, un-strand multi-user auth in the TS client, and clear the release-hygiene backlog. + +**Architecture:** All engine fixes are already merged-ready on PR #81. This plan is release engineering plus one contained TS client feature (send the optional username the server protocol already accepts). + +**Tech Stack:** Rust workspace (six published crates), TS client (Node ≥22, pnpm, zero deps), GitHub Actions release.yml, ghcr.io, crates.io, npm. + +**Hard rules:** Never push to main (branch + PR only). Never rebaseline `baseline/main.json` from a laptop. Smoke-test release artifacts with the documented flow BEFORE publishing (v0.4.1/0.4.2 shipped a P0 because gates were self-referential). + +--- + +### Task 0: Merge PR #81 (BLOCKS EVERYTHING) + +- [ ] **Step 1:** Kirby reviews + approves https://github.com/ZVN-DEV/powdb/pull/81 (one approval merges the whole sweep: 2 security fixes + 2 correctness fixes + ~50 doc fixes). +- [ ] **Step 2:** Merge via GitHub (squash or merge per repo convention). Confirm required checks green first. +- [ ] **Step 3:** `git checkout main && git pull` locally. + +### Task 1: TS client — send username for multi-user auth + +The server's Connect frame already carries an optional username (`crates/server/src/protocol.rs:38-41`: length-prefixed, appended after the password field; `crates/server/src/handler.rs` rejects missing usernames when users are defined). The TS client never sends it, so multi-user servers reject every Node client. + +**Files:** +- Modify: `clients/ts/src/protocol.ts` (Connect message variant + encoder, ~line 34 and ~line 56) +- Modify: `clients/ts/src/index.ts` (`ClientOptions` ~line 38, handshake call site) +- Modify: `clients/ts/src/pool.ts` (pass-through option) +- Test: `clients/ts/test/protocol.test.ts`, `clients/ts/test/client.test.ts` + +- [ ] **Step 1: Branch** — `git checkout -b feat/ts-client-user-auth` +- [ ] **Step 2: Write failing protocol tests** in `clients/ts/test/protocol.test.ts`, matching the existing `test(name, fn)` style: + +```ts +await test("encodes Connect with username after password", async () => { + const buf = encodeMessage({ type: "Connect", dbName: "main", password: "pw", username: "alice" }); + const decoded = decodeMessage(buf); // round-trip + assert.equal(decoded.username, "alice"); +}); +await test("encodes Connect with null username as legacy frame", async () => { + const buf = encodeMessage({ type: "Connect", dbName: "main", password: "pw", username: null }); + // must be byte-identical to the pre-username frame so old servers accept it +}); +``` + +- [ ] **Step 3:** Run `pnpm test` in `clients/ts` — expect the two new tests to FAIL (unknown field `username`). +- [ ] **Step 4: Implement protocol change** — extend the Connect variant in `protocol.ts`: + +```ts +| { type: "Connect"; dbName: string; password: string | null; username: string | null } +``` + +Encoder: when `username !== null`, append `encodeString(username)` after the password field (mirror the server: it reads username only `if pos < payload.len()`, so omitting the field entirely when null preserves byte-compat with 0.4.x servers and the legacy frame shape). Decoder: parse optional trailing username the same way the server does. + +- [ ] **Step 5:** Add `user?: string` to `ClientOptions` in `index.ts`; thread to the handshake (`username: opts.user ?? null`). Same pass-through in `pool.ts` options. +- [ ] **Step 6:** Run `pnpm test` — protocol tests PASS. +- [ ] **Step 7: Live integration test** — start `target/release/powdb-server` (from merged main) with a data dir seeded via `powdb-cli useradd alice --role readwrite --password s3cret` and `useradd bob --role readonly --password hunter2`. Test in `client.test.ts` style: connect as alice → insert OK; connect as bob → select OK, insert rejected with `permission denied` PowDBError (NOT a crash); connect with no user → `auth_failed`. (Server must be ≥0.4.6 for the readonly-rejection assertions.) +- [ ] **Step 8: Tarball hygiene (rides along):** set `"declarationMap": false, "sourceMap": false` in `clients/ts/tsconfig.json` (kills the 12 dead `.map` files contradicting the 0.3.4 changelog claim). Run `npm pack --dry-run` — confirm no `.map` files. +- [ ] **Step 9: Version + docs.** Bump `clients/ts/package.json` to **0.4.0** (semver-minor for a feature in 0.x; also intentionally aligns the client's minor with the server's — note in CHANGELOG). Update `clients/ts/CHANGELOG.md` (username support, map-file removal, compat table: multi-user mode now requires client ≥0.4.0 AND server ≥0.4.6 for enforced roles) and `clients/ts/README.md` (document `user` option with example; remove the multi-user incompatibility caveat added 2026-06-09, replacing it with the version matrix). +- [ ] **Step 10:** Full gate: `pnpm run build && pnpm test` (all suites). Commit: `feat(ts-client): send username for multi-user auth; drop dead source maps (0.4.0)`. +- [ ] **Step 11:** PR → review → merge. **Do not `npm publish` until Task 2's v0.4.6 server is live** (the integration assertions reference enforced roles). + +### Task 2: Cut v0.4.6 + +**Files:** +- Modify: root `Cargo.toml` (workspace version → 0.4.6), `Cargo.lock` +- Modify: `CHANGELOG.md` (promote `[Unreleased]` → `[0.4.6] - `) +- Modify: `SECURITY.md` (supported-versions: add 0.4.6 row, 0.4.5 → superseded; the "ships in the next release" readonly note → "enforced as of 0.4.6") +- Modify: `RELEASES.md` checklist if any step drifted + +- [ ] **Step 1:** Branch `release/0.4.6` from updated main. Bump workspace version; `cargo build --workspace` to refresh the lockfile. +- [ ] **Step 2:** CHANGELOG: rename `[Unreleased]` section to `[0.4.6] - `; add fresh empty `[Unreleased]`. +- [ ] **Step 3:** SECURITY.md edits above. Sanity-grep: `grep -rn "0\.4\.5" README.md docs/ site/ examples/` — bump user-facing version pins (README install line, powdb-vs-sqlite pin, aws-ecs image tag, deploy README docker tag) to 0.4.6. +- [ ] **Step 4:** Full gate: `cargo test --workspace && cargo clippy --workspace --all-targets -- -D warnings && cargo fmt --all --check`. +- [ ] **Step 5: Pre-publish smoke (MANDATORY, non-self-referential):** `cargo build --release`, then with the built binaries in a fresh temp dir run, exactly as documented in README/getting-started: the full PowQL flow (type/insert/filter/group/transactions), kill -9 + restart WAL replay (all rows recovered), **both attack repros** (oversized insert → clean error + server alive; readonly user → writes denied), backup → restore roundtrip. +- [ ] **Step 6:** PR `release/0.4.6` → approve → merge. Tag `v0.4.6` on main; push tag. Confirm `release.yml` builds binaries + pushes `ghcr.io/zvn-dev/powdb:v0.4.6` + `latest`. +- [ ] **Step 7: Publish to crates.io in dependency order:** `powdb-storage` → `powdb-auth` → `powdb-query` → `powdb-backup` → `powdb-server` → `powdb-cli` (per RELEASES.md; wait for each index propagation). +- [ ] **Step 8: Post-publish verification:** on a clean machine/dir, `cargo install powdb-cli --version 0.4.6` and re-run the Step-5 smoke against the *installed* binary. Then `npm publish` the TS client 0.4.0 (Task 1) and re-run its integration script against the published package. +- [ ] **Step 9:** GitHub release notes from CHANGELOG; verify crates.io pages render READMEs and the homepage link works (it was a 404 until this week). + +### Task 3: ghcr package visibility + +- [ ] **Step 1:** GitHub → zvn-dev org → Packages → `powdb` → Package settings → Change visibility → **Public**. (Leftover from the org transfer; the new package defaulted private.) +- [ ] **Step 2:** Verify logged-out: `docker logout ghcr.io && docker pull ghcr.io/zvn-dev/powdb:v0.4.6` succeeds (or `curl -s https://ghcr.io/v2/zvn-dev/powdb/tags/list` returns 401-with-public-token flow rather than 404). + +### Task 4: Hygiene backlog + +- [ ] **Step 1:** Review + merge the 5 open dependabot PRs (batch-merge after CI; rebase any that conflict with the sweep merge). +- [ ] **Step 2:** Depot bench migration: install the Depot GitHub app (Kirby, manual), then merge `chore/bench-depot-runner`, run `gh workflow run bench.yml` on the Depot runner, and **rebaseline `baseline/main.json` from that Depot run only** (never from a laptop). +- [ ] **Step 3:** Kirby runs `! flyctl auth login`, then `fly apps list` — confirm no stray PowDB apps (expected: none; fly.toml was always a template). +- [ ] **Step 4:** Delete local test residue now that the sweep is merged: `rm -rf powdb_data clients/ts/powdb_data node_modules` at repo root (all gitignored; root `node_modules` is a stale MCP cache, the real one lives in `clients/ts/`). + +### Exit criteria + +- v0.4.6 live on crates.io (6 crates), ghcr (public), GitHub releases — verified from a clean install. +- TS client 0.4.0 on npm; a Node client can authenticate to a multi-user server and readonly is enforced end-to-end. +- Zero open dependabot PRs; bench gate running on Depot; no stray deployments anywhere. diff --git a/examples/deploy/README.md b/examples/deploy/README.md index acb07ae..c8125e2 100644 --- a/examples/deploy/README.md +++ b/examples/deploy/README.md @@ -28,10 +28,20 @@ Notes: - `min_machines_running = 1` keeps the database always-on; `auto_stop_machines` is `false` so Fly never suspends a stateful service. -## Docker / Compose +## Docker -See [`docker-compose.yml`](https://github.com/zvndev/powdb/blob/main/docker-compose.yml) -in the repo root for a local-only quick-start. +Quick-start with the published image (note: the repo-root `docker-compose.yml` +is the benchmark harness — it does not define a PowDB service): + +```bash +docker run -d --name powdb \ + -p 5433:5433 \ + -v powdb_data:/data \ + -e POWDB_DATA=/data \ + -e POWDB_BIND=0.0.0.0 \ + -e POWDB_PASSWORD=change-me \ + ghcr.io/zvn-dev/powdb:v0.4.5 +``` ## AWS ECS Fargate + EFS diff --git a/examples/deploy/fly.toml b/examples/deploy/fly.toml index 944cf48..43eb313 100644 --- a/examples/deploy/fly.toml +++ b/examples/deploy/fly.toml @@ -22,6 +22,8 @@ primary_region = "iad" RUST_LOG = "info" POWDB_DATA = "/data" POWDB_PORT = "5433" + # Bind all interfaces — the server defaults to 127.0.0.1, which Fly's proxy can't reach. + POWDB_BIND = "0.0.0.0" # Phase 1 hardening: refuse to start when a password is set without TLS. # Leave OFF until you've mounted certs via secrets (POWDB_TLS_CERT / # POWDB_TLS_KEY) or fronted the service with a TLS-terminating proxy. diff --git a/examples/deploy/railway/README.md b/examples/deploy/railway/README.md index 945a072..c6c376a 100644 --- a/examples/deploy/railway/README.md +++ b/examples/deploy/railway/README.md @@ -45,9 +45,11 @@ The host/port come from the TCP proxy you configured in step 4. are pinned to a region; if Railway moves your service across regions, you'll need to snapshot and restore manually. Pin the region in the dashboard for production. -- **No managed backups.** Schedule a sidecar `cron` that runs - `powdb-cli` exports against the volume, or snapshot the volume via - Railway's API on a schedule. +- **No managed backups.** Use `powdb-cli backup` (full / incremental, + plus coarse PITR on restore) against the volume — but note backups are + **offline**: stop the server first, so schedule them in a maintenance + window or snapshot the volume via Railway's API instead. See + [docs/backup-and-restore.md](../../../docs/backup-and-restore.md). - **Bandwidth pricing.** Heavy benchmark traffic over the TCP proxy bills egress. For benchmark runs, use the Fly.io or AWS examples — Railway is best for developer-friendly persistent deploys, not for hammering. diff --git a/site/getting-started.html b/site/getting-started.html index 263e83b..cccc9b2 100644 --- a/site/getting-started.html +++ b/site/getting-started.html @@ -19,7 +19,7 @@
Getting Started PowQL Reference - + GitHub @@ -39,8 +39,8 @@

1Install

cargo install powdb-cli
 cargo install powdb-server
-

Or build from source (requires Rust 1.80+; install from rustup.rs if needed):

- 
git clone https://github.com/zvndev/powdb.git
+    
Or build from source (requires Rust 1.93+; install from rustup.rs if needed):

+    
git clone https://github.com/ZVN-DEV/powdb.git
 cd powdb
 cargo build --release

 
@@ -55,7 +55,7 @@ 
2Start the REPL

 cargo run --release -p powdb-cli

You should see:

-
PowDB v0.2.0 -- embedded mode +
PowDB v0.4.5 -- embedded mode Data directory: ./powdb_data Type PowQL queries. Use Ctrl-D to exit. @@ -188,8 +188,18 @@

10Delete

To delete all rows in a table: User delete

- -

11Server Mode

+ +

11Transactions

+

Statements run in autocommit by default -- each one is durable on its own. For atomic multi-statement changes (and much faster bulk loads), use explicit transactions:

+ 
powql> begin
+powql> insert User { name := "Alice", email := "alice@example.com", age := 30 }
+powql> insert User { name := "Bob", email := "bob@example.com", age := 25 }
+powql> commit
+

Everything between begin and commit applies atomically -- either all statements succeed or none do. Use rollback instead of commit to discard uncommitted changes. Batching writes in a transaction shares a single WAL fsync at commit, which makes bulk loads roughly 50x faster with identical durability.

+

Full details: Transactions in the PowQL reference.

+ + +

12Server Mode

For multi-client access, run PowDB as a server.

Start the server

@@ -198,9 +208,9 @@

Start the server

Connect from a client

powdb-cli --remote localhost:5433
-
PowDB v0.2.0 -- remote mode +
PowDB v0.4.5 -- remote mode Connecting to localhost:5433 ... -Connected to db `main` (server v0.2.0) +Connected to db `main` (server v0.4.5) Type PowQL queries. Use Ctrl-D to exit. powql>
@@ -221,6 +231,16 @@

Password authentication

# Connect with the password powdb-cli --remote localhost:5433 --password mysecret +

Multi-user authentication

+

PowDB also supports named users with roles (admin, readwrite, readonly), stored as argon2id hashes in the data directory. Once any user is defined, the shared password is no longer used:

+ 
# Create users (role defaults to readwrite)
+powdb-cli --data-dir ./powdb_data useradd alice --role admin --password s3cret
+powdb-cli --data-dir ./powdb_data useradd bob --role readonly --password hunter2
+
+# Connect as a named user
+powdb-cli --remote localhost:5433 --user alice --password s3cret
+

For containerized first boots, set POWDB_ADMIN_USER and POWDB_ADMIN_PASSWORD and the server bootstraps an admin user on startup -- no CLI needed. See Multi-user authentication for the full workflow.

+

What's Next

This tutorial covered tables, inserts, queries, aggregates, updates, indexes, deletes, and server mode. PowDB supports much more, including joins, group by, subqueries, materialized views, window functions, and set operations.

diff --git a/site/index.html b/site/index.html index 64fb9cd..66ab9d3 100644 --- a/site/index.html +++ b/site/index.html @@ -20,9 +20,9 @@ Getting Started PowQL Reference Benchmarks - Releases - Changelog - + Releases + Changelog + GitHub @@ -40,7 +40,7 @@

A database engine that's
3-10x faster than

Get Started - View on GitHub + View on GitHub
$ cargo install powdb-cli @@ -52,7 +52,7 @@

A database engine that's
3-10x faster than

Benchmarks

-

PowDB vs SQLite on 100K rows. Both engines in memory-mode. Real numbers, no cherry-picking.

+

PowDB vs SQLite on 100K rows. Both engines in memory-mode. All 14 workloads — real numbers, no cherry-picking.

@@ -117,6 +117,36 @@

Benchmarks

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
282ns 3.0x
multi_col_and_filter2.22ms4.70ms2.1x
insert_batch_1k238ns320ns1.3x
delete_by_filter1.76ms2.35ms1.3x
scan_filter_project_top1009.6µs12.7µs1.3x
point_lookup_nonindexed350µs432µs1.2x
@@ -221,7 +251,7 @@

Compiled Predicates

B+ Tree Indexes

-

Disk-persisted B+ tree indexes in a custom BIDX binary format. Point lookups resolve in under 150ns. Indexes survive restarts and are used automatically.

+

Disk-persisted B+ tree indexes in a custom BIDX binary format. Indexed point lookups resolve in under 100ns. Indexes survive restarts and are used automatically.

WAL + Crash Recovery

@@ -237,7 +267,7 @@

TypeScript Client

TLS + Authentication

-

Production-ready security with TLS encryption and password authentication. Set POWDB_PASSWORD to lock down your server.

+

TLS encryption plus two auth modes: a shared password (POWDB_PASSWORD) or named users with roles (admin / readwrite / readonly, argon2id-hashed) shipped in v0.4.5.

Zero C Dependencies

@@ -251,6 +281,21 @@

Pipeline Query Language

+ +
+
+

When PowDB is not the right fit

+

PowDB is pre-1.0 and deliberately scoped. Be honest with yourself before adopting it:

+
    +
  • Pre-1.0 format stability -- the on-disk format may shift across minor versions. Pin versions and expect to re-import on upgrades until 1.0.
    • +
  • No SQL compatibility -- PowQL is a different language by design. No ORMs, BI dashboards, JDBC drivers, or DB browsers will speak to it.
    • +
  • Single-node only -- no replication, sharding, or consensus.
    • +
  • No MVCC, no online backup -- single writer with parallel readers; backups (full, incremental, PITR) are offline-only today.
    • +
+

If any of those are dealbreakers, use SQLite -- and we say so plainly in PowDB vs SQLite: when to use which.

+
+
+