Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,13 @@ The measurable result: 3–10× faster than SQLite on aggregate and scan workloa
- **Embedded / edge / serverless workloads** where query latency is a tight budget and you don't want SQLite's quirks.
- **Single-node analytics** over tables that fit on disk. The scan path is zero-syscall (mmap) and filters are compiled to byte-level predicates.
- **You control both sides** (the DB and the app). PowDB has no Postgres wire protocol, no ODBC, no legacy compatibility. The client is a TCP binary protocol or an in-process Engine.
- **You want to read the code.** Four crates, ~20K lines, no generated parsers, no plan-language IR.
- **You want to read the code.** Eight crates, ~45K lines, no generated parsers, no plan-language IR.

### When it's *not* the right choice

- You need a drop-in Postgres/MySQL replacement. PowQL is a different language; there is **no SQL compatibility layer**, and that is a deliberate design decision (the translation tier is the thing we're removing).
- You need multi-node replication, sharding, or Raft-style consensus. Single-node only today.
- You need fine-grained ACLs, row-level security, or multi-tenant isolation. Auth is a single shared password.
- You need fine-grained ACLs, row-level security, or multi-tenant isolation. Auth is named users with coarse roles (admin/readwrite/readonly) since 0.4.5 — shared-password mode still available — but nothing per-table or per-row.
- You need user-defined functions, stored procedures, or triggers.

---
Expand Down Expand Up @@ -109,14 +109,15 @@ Compare SQL: `SELECT name, age FROM User WHERE age > 25 ORDER BY age DESC LIMIT
| `AND`, `OR`, `NOT` | `and`, `or`, `not` (lowercase) |
| `User.posts` (link navigation) | not yet implemented |
| `let x := ...` | not yet implemented |
| `count: count(.name)` (aggregate keyword as alias) | fails `expected alias name` — `sum:` fails too; use `n:`, `cnt:`, `total:` |

---

## Type system

Canonical type names: `str`, `int`, `float`, `bool`, `datetime`, `uuid`, `bytes`.

**Footgun:** the executor's type resolver falls back to `TypeId::Str` for any unknown name (`crates/query/src/executor.rs`), so `string`, `varchar`, or a typo silently produces a Str column with no error. Always use the canonical names above.
**Footgun:** the executor's type resolver falls back to `TypeId::Str` for any unknown name (`crates/query/src/executor/`), so `string`, `varchar`, or a typo silently produces a Str column with no error. Always use the canonical names above.

`required` is a prefix keyword on the field, not a `!` suffix: `required name: str`, never `name: str!`.

Expand All @@ -128,7 +129,7 @@ These are the design moves that buy the speedup. Understanding them keeps you fr

1. **Planner is a pure function.** It does not touch the catalog — it emits `RangeScan` speculatively, and the executor lowers to `Filter(SeqScan)` at runtime if no index exists. This keeps the parser → plan pipeline allocation-free for cache hits.
2. **Plan cache hashes canonical PowQL.** Literals are substituted at lookup time (FNV-1a hash, `crates/query/src/plan_cache.rs`). A repeated `User filter .id = <N>` reuses the same plan for all N.
3. **Compiled integer predicates.** `Filter(SeqScan)` on simple numeric predicates compiles into a branch-free byte-level check that skips full row decoding. See `execute_plan` fast paths in `crates/query/src/executor.rs`.
3. **Compiled integer predicates.** `Filter(SeqScan)` on simple numeric predicates compiles into a branch-free byte-level check that skips full row decoding. See `execute_plan` fast paths in `crates/query/src/executor/` (module dir).
4. **mmap-based scans.** The storage layer exposes `try_for_each_row_raw` over memory-mapped heap files. Early termination is a `return ControlFlow::Break`.
5. **Slotted 4KB pages + persistent B+tree indexes.** Standard, but the index format (BIDX, binary) is crash-safe and survives restart with no rebuild.
6. **WAL with group commit at statement boundaries.** Writes are durable by default; throughput is maintained by batching.
Expand Down Expand Up @@ -157,6 +158,8 @@ cargo run --release -p powdb-cli # embedded REPL
cargo run --release -p powdb-cli -- --remote host:5433 --password <pw>
```

**The REPL is line-oriented.** A statement split across lines fails to parse — write each statement on one line.

### TCP server

```bash
Expand Down Expand Up @@ -201,19 +204,17 @@ Return shapes:

## What's shipped vs. what's planned

Shipped: joins (inner/left/right/cross, nested-loop + hash), GROUP BY + HAVING, DISTINCT, UNION / UNION ALL, subqueries (IN, EXISTS, correlated), CASE, LIKE, BETWEEN, IN-list, window functions (ROW_NUMBER, RANK, DENSE_RANK, SUM/AVG/COUNT/MIN/MAX over partition), arithmetic, string/math/datetime scalars, CAST, COALESCE, materialized views with auto-refresh, upsert, prepared queries with literal substitution, explicit transactions (`begin` / `commit` / `rollback`), password auth, TLS (`POWDB_TLS_CERT` / `POWDB_TLS_KEY`), WAL + crash recovery, persistent indexes.
Shipped: joins (inner/left/right/cross, nested-loop + hash), GROUP BY + HAVING, DISTINCT, UNION / UNION ALL, subqueries (IN, EXISTS, correlated), CASE, LIKE, BETWEEN, IN-list, window functions (ROW_NUMBER, RANK, DENSE_RANK, SUM/AVG/COUNT/MIN/MAX over partition), arithmetic, string/math/datetime scalars, CAST, COALESCE (`??`), materialized views with auto-refresh, upsert, multi-row INSERT, prepared queries with literal substitution, explicit transactions (`begin` / `commit` / `rollback`), password auth + multi-user auth (named users, admin/readwrite/readonly roles), TLS (`POWDB_TLS_CERT` / `POWDB_TLS_KEY`), WAL + crash recovery, persistent indexes, backup/restore (full/incremental/PITR, offline).

Planned (design doc only — don't use): link navigation (`User.posts`), `let` bindings, default operator (`??`), UDFs, per-row permissions, replication.
Planned (design doc only — don't use): link navigation (`User.posts`), `let` bindings, UDFs, per-row permissions, replication.

---

## For contributors

Build: `cargo build --workspace`. Test: `cargo test --workspace`. Lint: `cargo clippy --workspace --all-targets -- -D warnings`. Format: `cargo fmt --all`.

CI gates on `main`:
- `clippy + fmt + test` — `.github/workflows/ci.yml`
- `criterion + regression gate` — `.github/workflows/bench.yml` (blocks merges if any of 7 load-bearing workloads regress >7% against the checked-in baseline)
CI gates on `main` (all in `.github/workflows/ci.yml`): clippy/fmt/test (2-OS matrix), miri, asan, cargo audit, MSRV consistency, examples-smoke. `.github/workflows/bench.yml` (criterion + regression gate) is **manual-only** (`workflow_dispatch`) and NOT a merge gate — run the gate locally instead (see above).

Internal docs:
- `CLAUDE.md` — codebase guide for Claude Code (architecture, crate graph, common patterns)
Expand Down
39 changes: 39 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed
- **Oversized rows no longer kill the server (remote DoS).** Inserting or
updating a row whose encoded size exceeds one 4 KB page (`MAX_ROW_DATA_SIZE`,
4070 bytes) previously hit a `panic!` in the heap layer, which — combined with
`panic = "abort"` — terminated the entire server process. Any connected client
could take down the database with a single ~5 KB string insert. The heap now
rejects oversized rows with a graceful `row too large: N bytes exceeds max M
bytes` query error before anything is written or WAL-logged (an oversized
update can no longer poison WAL replay), and the connection and server keep
running. There is still no large-object/overflow-page support — values over
~4 KB are rejected, not stored.
- **`readonly` role is now enforced at the query layer.** Previously the role
was authenticated and stored but never checked: a `readonly` user could
insert, update, delete, and drop tables. The server now classifies each
parsed statement and rejects writes (DML, DDL, view DDL, and transaction
control) from `readonly` principals with `permission denied: role 'readonly'
cannot execute write statements`. Unknown roles fail closed. Shared-password
mode, open mode, and embedded use are unaffected.
- **NULL values now arrive as `null` on the wire instead of `{}`.** The server
serialized SQL NULLs as `{}`, which the remote CLI displayed verbatim and
which broke the TS client's documented `"null"` sentinel for typed-row
decoding. The wire serialization is now the bareword `null`, and the remote
REPL renders it as `NULL`, matching embedded mode.
- **Window aggregates without `order` now compute the whole-partition value.**
`avg(.x) over (partition .d)` previously returned a running aggregate per
row (frame = partition start → current row) even with no `order` clause; per
standard semantics the frame is now the entire partition. Ordered windows
keep the running-frame behavior; ranking functions are unchanged.

### Documentation
- Ecosystem-wide accuracy sweep: site pages synced to v0.4.5 (banners were
v0.2.0, MSRV corrected to 1.93), crates.io homepage URL fixed (was a 404),
README/CONTRIBUTING/AGENTS no longer claim the bench suite is a CI merge
gate, SECURITY.md documents both auth modes and the ≤0.4.5 readonly caveat,
RELEASES.md covers all six crates + the Docker image, deploy examples fixed
(`fly.toml` was missing `POWDB_BIND=0.0.0.0`), TS client docs document the
multi-user-server incompatibility, and AGENTS.md gained small-model-tested
gotchas (reserved aggregate keywords as aliases; line-oriented REPL).

## [0.4.5] - 2026-06-09

### Added
Expand Down
7 changes: 5 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ PowDB is a from-scratch database engine with its own query language (PowQL). No
powdb-cli ──→ powdb-server ──→ powdb-query ──→ powdb-storage
↑ ↑
powdb-bench powdb-compare

powdb-auth ←── powdb-server, powdb-cli (user store + roles; no inter-crate deps)
powdb-backup ←── powdb-cli (depends on powdb-storage + powdb-query)
```

### Query Pipeline
Expand All @@ -30,7 +33,7 @@ PowQL text → Lexer (token stream) → Parser (AST) → Planner (PlanNode tree)
- **Lexer** (`crates/query/src/lexer.rs`): Tokenizes PowQL input
- **Parser** (`crates/query/src/parser.rs`): Recursive descent, produces `Statement` AST
- **Planner** (`crates/query/src/planner.rs`): Pure function (no catalog access), produces `PlanNode` tree. Speculatively emits `RangeScan` for range inequalities
- **Executor** (`crates/query/src/executor.rs`): Runs plans against the storage engine. Has fast paths for common patterns (count, project+limit, sort+limit, agg, update, delete). Lowers `RangeScan` → `Filter(SeqScan)` at runtime when no index exists
- **Executor** (`crates/query/src/executor/` module dir): Runs plans against the storage engine. Has fast paths for common patterns (count, project+limit, sort+limit, agg, update, delete). Lowers `RangeScan` → `Filter(SeqScan)` at runtime when no index exists
- **Plan Cache** (`crates/query/src/plan_cache.rs`): FNV-1a hash, stores canonical plans, substitutes literals at lookup time

### Storage Engine
Expand Down Expand Up @@ -81,7 +84,7 @@ cargo run -p powdb-bench --bin compare
3. Add parser production to `crates/query/src/parser.rs`
4. Add plan node (if needed) to `crates/query/src/plan.rs`
5. Add planner case to `crates/query/src/planner.rs`
6. Add executor case to `crates/query/src/executor.rs`
6. Add executor case to `crates/query/src/executor/` (start in `mod.rs` / `plan_exec.rs`)

### Adding an executor fast path
Fast paths match on specific `PlanNode` shapes in `execute_plan()`. Pattern-match the plan tree and handle it before the generic recursive executor. Always verify with benchmarks.
Expand Down
10 changes: 7 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ cargo run --release -p powdb-compare # wide bench vs SQLite + Postgres (add --f
```
crates/storage # slotted pages, B+ tree, WAL, buffer pool, catalog
crates/query # lexer, parser, planner, executor (Engine)
crates/auth # user store, roles, argon2id password hashing
crates/backup # offline backup/restore (full, incremental, PITR)
crates/server # Tokio TCP server + binary wire protocol
crates/cli # rustyline REPL (embedded + remote mode)
crates/bench # criterion benchmarks + regression gate
Expand All @@ -49,7 +51,7 @@ clients/ts # TypeScript client + demo
### Branch protection on `main`

- PRs are required (no direct pushes)
- 7 status checks must pass: clippy + fmt + test (x2 OS matrix), miri, asan, audit, MSRV consistency, and the bench regression gate
- 7 status checks must pass, all from `ci.yml`: clippy + fmt + test (x2 OS matrix), miri, asan, audit, MSRV consistency, and examples-smoke
- Force-push is rejected by branch protection

Admin bypass exists for break-glass scenarios (security patches, recovering from a broken state). **Do not use it for routine work** — routine work goes through PRs even when bypass is technically available.
Expand All @@ -74,11 +76,13 @@ PRs must pass these gates (see `.github/workflows/`):
- **asan** — AddressSanitizer run
- **audit** — `cargo audit` against the advisory database
- **msrv-consistency** — verifies the declared MSRV (`1.93`) builds
- **criterion + regression gate** — benchmark must not regress beyond thresholds (`.github/workflows/bench.yml`)
- **examples-smoke** — terraform validate + compose config + dev.sh cycle on the deploy examples

The criterion benchmark suite (`.github/workflows/bench.yml`) is **manual-only** (`workflow_dispatch`) and is *not* a required PR gate — shared-runner noise makes it unreliable as a blocking check. Run the regression gate locally instead (below).

## Benchmark Regression Gate

The criterion gate compares each workload's median against baselines in `crates/bench/baseline/main.json`. Thresholds vary by workload (7-20%).
The criterion gate compares each workload's median against baselines in `crates/bench/baseline/main.json`. Thresholds vary by workload (7-20%). Run it locally with `cargo bench -p powdb-bench && cargo run -p powdb-bench --bin compare`, or on demand in CI with `gh workflow run bench.yml`.

If you intentionally change performance characteristics:
```bash
Expand Down
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ edition = "2021"
rust-version = "1.93"
license = "MIT"
repository = "https://github.com/zvndev/powdb"
homepage = "https://zvndev.github.io/powdb/"
homepage = "https://zvn-dev.github.io/powdb/"
readme = "README.md"

[workspace.dependencies]
Expand Down
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# PowDB

[![CI](https://github.com/zvndev/powdb/actions/workflows/ci.yml/badge.svg)](https://github.com/zvndev/powdb/actions/workflows/ci.yml)
[![bench](https://github.com/zvndev/powdb/actions/workflows/bench.yml/badge.svg)](https://github.com/zvndev/powdb/actions/workflows/bench.yml)
[![crates.io](https://img.shields.io/crates/v/powdb-cli.svg)](https://crates.io/crates/powdb-cli)
[![docs.rs](https://img.shields.io/docsrs/powdb-query)](https://docs.rs/powdb-query)
[![MSRV](https://img.shields.io/badge/MSRV-1.93-blue)](https://github.com/zvndev/powdb/blob/main/Cargo.toml)
Expand Down Expand Up @@ -30,7 +29,7 @@ Evaluating PowDB? Start with the honest comparison: [PowDB vs SQLite -- when to
| Filter + project | `SELECT name, age FROM User WHERE age > 25` | `User filter .age > 25 { .name, .age }` |
| Sort + limit | `SELECT * FROM User ORDER BY age DESC LIMIT 10` | `User order .age desc limit 10` |
| Aggregate with filter | `SELECT AVG(age) FROM User WHERE city = 'NYC'` | `avg(User filter .city = "NYC" { .age })` |
| Group + having | `SELECT status, COUNT(*) FROM User GROUP BY status HAVING COUNT(*) > 5` | `User group .status having count(*) > 5 { .status, count(*) }` |
| Group + having | `SELECT status, COUNT(name) FROM User GROUP BY status HAVING COUNT(name) > 5` | `User group .status having count(.name) > 5 { .status, n: count(.name) }` |

PowQL uses `.field` dot syntax for column references, `:=` for assignments, and `"double quotes"` for strings. The pipeline reads like a sentence: *"User, filter age greater than 25, order by name, limit 10, give me name and age."*

Expand All @@ -43,7 +42,7 @@ Full language reference: [docs/POWQL.md](https://github.com/zvndev/powdb/blob/ma
cargo install powdb-cli
cargo install powdb-server

# TypeScript client (Node 18+)
# TypeScript client (Node 18+) — versions independently of the server crates (currently 0.3.x)
npm install @zvndev/powdb-client

# Prebuilt binaries (linux x86_64, macos aarch64)
Expand Down Expand Up @@ -201,10 +200,14 @@ if (result.kind === "rows") console.table(result.rows);
| Variable | Default | Description |
|---|---|---|
| `POWDB_PORT` | `5433` | TCP port for the server |
| `POWDB_BIND` | `127.0.0.1` | Interface to bind; set `0.0.0.0` behind a platform proxy (Fly, Railway) |
| `POWDB_DATA` | `./powdb_data` | Data directory (heap files, WAL, catalog, indexes) |
| `POWDB_PASSWORD` | *(none)* | Shared password required on connect when no named users are defined (set as env var) |
| `POWDB_ADMIN_USER` / `POWDB_ADMIN_PASSWORD` | *(none)* | Bootstrap an `admin` user on startup when both are set and that user does not yet exist (password never logged) |
| `POWDB_TLS_CERT` / `POWDB_TLS_KEY` | *(none)* | Paths to PEM cert + key; when both are set the server serves TLS |
| `POWDB_REQUIRE_TLS` | *(off)* | When set (`1`/`true`), refuse to start if a password is configured without TLS |
| `POWDB_IDLE_TIMEOUT` | `300` | Seconds before an idle connection is closed |
| `POWDB_QUERY_TIMEOUT` | `30` | Per-query timeout in seconds |
| `POWDB_QUERY_MEMORY_LIMIT` | `268435456` | Per-query memory budget in bytes (256 MiB); over-budget queries error instead of OOM-killing the server |
| `RUST_LOG` | `info` | Log level (`debug`, `trace` for per-query timings) |

Expand Down Expand Up @@ -232,6 +235,7 @@ For a self-hostable starting point, see [`examples/deploy/fly.toml`](https://git
- Memory-mapped reads (zero-syscall scan path)
- Compiled integer predicates (branch-free filter at the byte level)
- Thread-safe concurrent reads via pread(2)/pwrite(2)
- Backup & restore: full + incremental + coarse point-in-time recovery (offline; `powdb-cli backup` / `restore` — see [docs/backup-and-restore.md](docs/backup-and-restore.md))

**Query engine**
- PowQL parser + planner + executor with plan cache (FNV-1a hashing, literal substitution)
Expand Down Expand Up @@ -273,6 +277,8 @@ For a self-hostable starting point, see [`examples/deploy/fly.toml`](https://git
crates/
storage/ Heap files, B+tree, WAL, catalog, page cache, row encoding
query/ Lexer, parser, planner, executor (Engine), plan cache
auth/ User store, roles, argon2id password hashing
backup/ Offline backup/restore (full, incremental, PITR)
server/ Tokio TCP server + binary wire protocol
cli/ Interactive REPL (embedded + remote modes)
bench/ Criterion benchmarks + regression gate
Expand All @@ -283,13 +289,15 @@ The engine is `powdb_query::executor::Engine`. It owns a `Catalog` (which owns `

## Benchmarks

PowDB has a CI-enforced regression gate that blocks PRs to `main` if any workload regresses beyond its threshold. Run locally:
PowDB has a benchmark regression gate that compares every workload against checked-in baselines. Run it locally before and after touching a hot path:

```bash
cargo bench -p powdb-bench # criterion suite (~60s)
cargo run --release -p powdb-bench --bin compare # regression gate
```

The gate also runs on-demand in CI via `workflow_dispatch` (`.github/workflows/bench.yml`) — it is **not** a required PR gate, because shared-runner noise makes it unreliable as a blocking check. The required PR gates live in `ci.yml`.

Run the PowDB vs SQLite comparison bench:

```bash
Expand Down
Loading
Loading