v3.0.0 PR

CHANGELOG.md

@@ -4,6 +4,151 @@ This file records notable project changes. It follows the
 [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format and uses
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.0.0] - 2026-08-04
+
+### BREAKING — Mandatory database migration
+
+This release pins DecentDB **v2.17.0** and bumps the on-disk database
+format from **13 → 14**. Every existing user database is now refused
+by the engine until migrated. On first open of a legacy file, Decent
+Bench offers a guided **in-place upgrade** that:
+
+1. Copies the source to a temp destination via the official
+   `decentdb-migrate` tool.
+2. Moves the original to `<name>.ddb.v13.bak` (preserved as the
+   explicit rollback handle).
+3. Carries the `.wal` and `.sync-journal` sidecars aside next to the
+   backup; excludes `.coord` so the engine rebuilds fresh
+   coordination state.
+4. Atomically swaps the temp destination into place.
+5. Restores the original from backup if any step fails.
+
+**The upgrade is one-way.** Older Decent Bench builds and older DecentDB
+releases will refuse to open the upgraded file. The backup is never
+deleted automatically; users must keep it until they have verified the
+new file.
+
+Headless `bin/headless_import.dart` and `bin/dbench_quality.dart` now
+emit an actionable `decentdb-migrate` invocation hint instead of just
+logging the failure.
+
+See `design/adr/0060-decentdb-2-17-format-14-guided-migration.md`.
+
+### Added — Tier 1 engine features
+
+- **Database performance profile:** New `[database_open]` TOML section
+  exposes `profile` (one of `default`, `low_memory`, `balanced`,
+  `embedded_fast`, `tuned_durable`), `plan_cache_enabled`, and
+  `plan_cache_max_bytes`. Critical ordering: `profile=` is emitted first
+  in the open-options string because selecting a profile resets the
+  entire `DbConfig`. See ADR-0062.
+- **Flush plan cache:** New Tools menu entry runs `PRAGMA flush_plan_cache`
+  against the open database.
+- **8 new `sys.*` operational metrics:** `sys.plan_cache`,
+  `sys.plan_cache_summary`, `sys.doctor_findings`, `sys.fix_plan`,
+  `sys.sync_shapes`, `sys.sync_shape_clients`,
+  `sys.sync_changeset_history`, `sys.sync_relay_sessions` (exact SQL
+  text, no `LIMIT`). The dead-code boundary short-circuit that masked
+  `sys.*` views was removed.
+
+### Added — Doctor / advisor panel
+
+- **Tools → Database Doctor** opens a new panel. Primary path: shell
+  out to `decentdb doctor --db <path> --format json --checks all
+  --include-recommendations=true`. Fallback: in-process
+  `sys.doctor_findings` + `sys.fix_plan` views; the fallback is
+  rendered with a prominent "Degraded results" banner so it cannot be
+  mistaken for a clean bill of health.
+- Forwarded CLI flags: `--verify-indexes`, `--verify-index <name>`,
+  `--max-index-verify`.
+- Non-zero `--fail-on=error` exit is treated as a normal unhealthy
+  database, not as a tool failure. See ADR-0061.
+
+### Added — EXPLAIN + ANALYZE
+
+- Parser now recognises multi-word operators added in v2.15-v2.17:
+  `HASH JOIN`, `INDEXED JOIN`, `STREAMING AGGREGATE`, `VIEW SCAN`,
+  `EXPANDED VIEW`.
+- Renders `est cost=N.NN` chips alongside `est rows` and `actual rows`.
+- `WorkspaceController.runAnalyze({tableName})` issues `ANALYZE` (or
+  `ANALYZE "<table>"` when scoped) and surfaces a success message.
+
+### Added — Schema browser
+
+- Indexes are now badged with per-kind icons: `Btree` (default),
+  `FullText` (manage-search), `Spatial` (public), `Trigram`
+  (text-fields). Non-Btree index kinds were previously indistinguishable.
+
+### Added — Maintenance actions
+
+- `WorkspaceController.saveAs(destPath)` invokes the engine `saveAs`
+  ABI for a compact copy of the open database.
+- `WorkspaceController.evictSharedWal(path)` is exposed but
+  refuses to run while the workspace is open (the engine documents
+  this call as unsafe with open handles).
+
+### Added — Full-text + index vocabulary
+
+- New SQL keywords in autocomplete and formatter: `FULLTEXT`, `BM25`,
+  `INDEXED`, `REBUILD`, `VERIFY`, `USING FULLTEXT`, `USING BTREE`,
+  `USING SPATIAL`, `USING TRIGRAM`, `ALTER INDEX`.
+- New SQL functions: `FULLTEXT_MATCH`, `BM25`, `BM25_SCORE`,
+  `FULLTEXT_RANK`.
+- `ALTER INDEX <name> VERIFY` and `ALTER INDEX <name> REBUILD` are
+  routed through the existing mutating SQL risk-assessment path.
+
+### Changed — Import fast path
+
+- Import pipelines in `import_execution_service.dart`,
+  `excel_import_support.dart`, `sqlite_import_support.dart`, and
+  `sql_dump_import_support.dart` now prefer
+  `Statement.executeBatchTyped(signature, rows)` (v2.16 API) when every
+  column in the table is expressible in the typed-batch signature
+  (`i`=INT64, `f`=FLOAT64, `t`=TEXT) and no column was observed to
+  contain null values (the v2.17 Dart binding rejects `null` for
+  typed-batch slots). Other column types continue to use the existing
+  per-row `bindAll` path.
+- A shared `typed_batch_classification.dart` module owns the column
+  classification so all importers stay in sync.
+- Throughput on a large `test-data/` fixture improved measurably
+  (single-pass C buffer allocation eliminates per-row Dart-to-C
+  marshalling). Existing import fixture matrix still passes
+  row-for-row identically.
+
+### Added — Export, schema, and workspace features
+
+- **Parquet Export:** Streaming cursor-based export to Parquet format
+  (`.parquet`) with schema fingerprint preservation and progress
+  indicator. Previously deferred from the v1.0.0 MVP "Next" list.
+- **Excel Export Enhancement:** Enhanced Office Open XML writer for
+  `.xlsx` result export with native type metadata preservation.
+- **Column Reordering:** Drag-and-drop column reordering in the results
+  grid with persistent per-tab state and reset-to-default functionality.
+- **Schema browser metadata parity with DecentDB v2.14.0:** Schema
+  browser now surfaces fields that the binding has always exposed but
+  the bridge was previously dropping: table row counts, primary-key
+  column lists, full foreign-key definitions (including composite
+  multi-column FKs), view `sqlText` and view dependency lists, covering
+  index `INCLUDE (...)` columns, index freshness flags, and per-column
+  `autoIncrement`. See ADR-0058.
+- **Structured DecentDB error diagnostics:** `BridgeFailure` now
+  extracts `subcode`, `retryable`, `permanent`, `sqlstate`, and
+  `docAnchor` directly from `DecentDbException.diagnostic`, and
+  translates `DecentDbAbiMismatchException` to `DDB_ERR_ABI_MISMATCH`
+  and `DecentDbNativeLoadException` to `DDB_ERR_NATIVE_LOAD`. See
+  ADR-0059.
+
+### Changed — Versioning
+
+- App version bumped to **3.0.0+1** (per `VERSIONING_GUIDE.md`: a change
+  that renders every existing user file unopenable without a
+  migration is a Major bump).
+- The 2.1.0 release line that was previously drafted here was never
+  shipped. Its intended additions (Parquet export, Excel-export
+  enhancements, column reordering, v2.14 schema-browser metadata parity
+  and structured error diagnostics) are consolidated into this v3.0.0
+  release.
+
 ## [2.0.0] - 2026-05-30
 
 ### Added
@@ -308,7 +453,8 @@ are documented here for traceability:
   metadata, bundled theme compatibility ranges, and project documentation with
   that release line.
 
-[unreleased]: https://github.com/sphildreth/decent-bench/compare/v2.0.0...HEAD
+[unreleased]: https://github.com/sphildreth/decent-bench/compare/v3.0.0...HEAD
+[3.0.0]: https://github.com/sphildreth/decent-bench/releases/tag/v3.0.0
 [2.0.0]: https://github.com/sphildreth/decent-bench/releases/tag/v2.0.0
 [1.1.0]: https://github.com/sphildreth/decent-bench/releases/tag/v1.1.0
 [1.0.0]: https://github.com/sphildreth/decent-bench/releases/tag/v1.0.0

README.md

@@ -19,7 +19,7 @@
     <img alt="License: Apache 2.0" src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square">
   </a>
   <img alt="Flutter desktop" src="https://img.shields.io/badge/Flutter-desktop-02569B?style=flat-square&logo=flutter&logoColor=white">
-  <img alt="DecentDB v2.8.0" src="https://img.shields.io/badge/DecentDB-v2.8.0-6f42c1?style=flat-square">
+  <img alt="DecentDB v2.17.0" src="https://img.shields.io/badge/DecentDB-v2.17.0-6f42c1?style=flat-square">
 </p>
 
 <p align="center">
@@ -64,14 +64,14 @@
 - ⚡ **Performance-Focused:** Background imports, paginated/streamed results grids, and best-effort query cancellation ensure the UI never freezes.
 - 🧭 **Rich Engine Metadata:** Schema browsing is powered by DecentDB's rich
    upstream schema snapshot (tables/views/indexes/triggers, checks, foreign keys,
-   generated columns, temp-object metadata, and canonical DDL), with v2.8.x
+   generated columns, temp-object metadata, and canonical DDL), with v2.17.x
   tooling metadata and query contracts used for schema fingerprints, parameter
   types, and result-column types.
-- 📊 **DecentDB v2.8 Diagnostics:** Database Statistics includes WAL, storage,
+- 📊 **DecentDB v2.17 Diagnostics:** Database Statistics includes WAL, storage,
   write-queue, sync, reactive, relay, process coordination, Lua extension
   inspection surfaces, plus rich structured error diagnostics, optional queued
   inline table edits, and a read-only local Web Console launcher.
-- 🧬 **Native Type Awareness:** DecentDB v2.8.x semantic and spatial types are
+- 🧬 **Native Type Awareness:** DecentDB v2.17.x semantic and spatial types are
   surfaced in schema details, result metadata, autocomplete, snippets, import
   type overrides, copy behavior, and CSV export display values.
 - 📊 **Diagnostics & Visualization:** Column statistics, database statistics,
@@ -94,7 +94,7 @@
 - 🪵 **Operational Visibility:** Open application logs from `Tools -> View Logs`. Structured JSON.CLEF log files are written per session to a configurable log directory (default `logs/` under the app config path).
 - 🧪 **Import Validation:** Blocking failure dialogs and richer import summaries make unsuccessful imports obvious and successful imports easier to verify.
 - 📤 **Typed Exports:** CSV, JSON, NDJSON, and Excel export stream result pages
-  and preserve DecentDB v2.8.x native value metadata where the format supports
+  and preserve DecentDB v2.17.x native value metadata where the format supports
   it. Result charts can be exported as PNG, and ERDs can be exported as PNG/JPG.
 - 📦 **Desktop Native:** Packaged for Linux, macOS, and Windows with a repeatable native-library staging helper.
 
@@ -186,7 +186,7 @@ Want to build from source or contribute? Welcome!
 
 Decent Bench pins the upstream Dart package by Git tag and expects the matching
 DecentDB desktop native library alongside it. CI and release packaging currently
-resolve `v2.8.0` from `apps/decent-bench/pubspec.lock` and download the matching
+resolve the pinned engine tag from `apps/decent-bench/pubspec.lock` and download the matching
 `decentdb-dart-native-<tag>-...` asset from
 [`sphildreth/decentdb` Releases](https://github.com/sphildreth/decentdb/releases).
 
@@ -266,6 +266,57 @@ Typical files under that root include:
 - 🧠 [`design/adr/README.md`](design/adr/README.md) — Architecture Decision Records
 - 🤖 [`AGENTS.md`](AGENTS.md) — Agent instructions and guardrails
 
+## 🩺 Troubleshooting
+
+### "DDB_ERR_TIMEOUT" while opening a database
+
+DecentDB acquires a process writer lock while opening a database (the
+`<db>.ddb.coord` sidecar file holds the lock state). If another
+DecentDB-backed process is still holding the lock, or if the `.coord`
+file is stale (left behind by a `kill -9`, crash, or improper close),
+or if the open path is on a slow filesystem (network mount, FUSE,
+encrypted volume), the engine waits up to
+`process_coordination_timeout_ms` (default 30s) and then returns
+`DDB_ERR_TIMEOUT`. The UI now surfaces a dialog explaining the
+cause and the remediation. Typical resolutions:
+
+1. Close any other DecentDB-backed process that might be holding the
+   writer lock.
+2. Remove a stale `<db>.ddb.coord` sidecar file (the engine rebuilds
+   it on the next open — it is not user data).
+3. Raise the engine's process-coordination wait in your `config.toml`:
+
+   ```toml
+   [database_open]
+   process_coordination_timeout_ms = 300000   # 5 minutes (engine side)
+   open_bridge_timeout_ms = 600000            # 10 minutes (bridge side)
+   ```
+
+   The **bridge timeout must be greater than the engine coordination
+   timeout** — otherwise the bridge will outrace the engine and surface a
+   misleading bridge-level timeout instead of the engine's actual
+   response. Defaults: engine 30s, bridge 5 min.
+
+   You can also override the bridge timeout at launch via the
+   `DECENT_BENCH_OPEN_TIMEOUT_MS` environment variable.
+
+### Sidecar files
+
+Each open `.ddb` file may have companion sidecars that are part of
+the on-disk format:
+
+| File | Purpose | Rebuildable? |
+| ---- | ------- | ------------ |
+| `<db>.ddb.wal` | Write-Ahead Log of pending changes | Yes (on close) |
+| `<db>.ddb.sync-journal` | Sync changeset journal | Yes (on close) |
+| `<db>.ddb.coord` | Process writer-lock state | Yes (on next open) |
+
+If you copy a `.ddb` for backup or transfer, copy all sidecars that
+are present. Removing a `.coord` sidecar is safe; removing a `.wal`
+or `.sync-journal` requires running DecentDB's built-in recovery on
+next open (the engine will replay what it can and rebuild state
+from the `.ddb`).
+
 ## 🤝 Contributing
 
 We love contributions! Before making non-trivial changes, please review the [`SPEC.md`](design/SPEC.md) and our [`AGENTS.md`](AGENTS.md) guidelines.

THIRD_PARTY_NOTICES.md

@@ -7,7 +7,7 @@ Apache 2.0 distribution. This file tracks attributions and license details.
 
 - `decentdb`
   - Version/source: Git dependency from `https://github.com/sphildreth/decentdb`,
-    path `bindings/dart/dart`, ref `v2.8.0`
+    path `bindings/dart/dart`, ref `v2.17.0`
   - License: Apache License 2.0
   - Upstream project: `https://github.com/sphildreth/decentdb`
 
@@ -21,7 +21,7 @@ Apache 2.0 distribution. This file tracks attributions and license details.
   - Copyright: Brendan Duncan
   - Source: `https://pub.dev/packages/archive`
 
-- `crypto` `3.0.6`
+- `crypto` `3.0.7`
   - License: MIT
   - Copyright: Dart project authors
   - Source: `https://pub.dev/packages/crypto`
@@ -51,7 +51,7 @@ Apache 2.0 distribution. This file tracks attributions and license details.
   - Copyright: Brendan Duncan
   - Source: `https://pub.dev/packages/image`
 
-- `sqlite3` `3.1.7`
+- `sqlite3` `3.5.1`
   - License: MIT
   - Copyright: Simon Binder
   - Source: `https://pub.dev/packages/sqlite3`

apps/decent-bench/README.md

@@ -23,7 +23,7 @@ Bench `2.0.0`, which builds on the project's shipped `1.0.0` MVP release.
   them into the normal generic or dedicated import path
 - desktop runner folders (`linux/`, `macos/`, `windows/`) are checked in
 - the DecentDB Dart package is pinned from the upstream Git tag
-   (`https://github.com/sphildreth/decentdb`), currently `v2.8.0`, and desktop
+   (`https://github.com/sphildreth/decentdb`), currently `v2.17.0`, and desktop
   packaging stages the matching `decentdb-dart-native-<tag>-...` release asset
   plus the official `decentdb-migrate` and `decentdb` CLI tools from the full
   release asset
@@ -43,15 +43,15 @@ Bench `2.0.0`, which builds on the project's shipped `1.0.0` MVP release.
 - schema browsing is backed by DecentDB's rich schema snapshot surface
   (`Schema.getSchemaSnapshot()`), including canonical DDL, checks, foreign keys,
   generated-column metadata, triggers, and temp-object metadata
-- DecentDB v2.8.x tooling metadata and query contracts flow through the bridge
+- DecentDB v2.17.x tooling metadata and query contracts flow through the bridge
   for schema fingerprints, parameter contracts, and result-column contracts
 - read-only ERD viewing uses the loaded schema snapshot to draw table nodes,
   foreign-key edges, missing-reference placeholders, search/filter context, and
   table-preview navigation without adding schema-design or mutation workflows
-- DecentDB v2.8.x native semantic/spatial types have first-class display
+- DecentDB v2.17.x native semantic/spatial types have first-class display
   helpers for schema details, result cells, autocomplete/snippets, import type
   overrides, WKB copy, and CSV export formatting
-- DecentDB v2.8.0 operational metrics, process coordination, queued writes,
+- DecentDB v2.17.0 operational metrics, process coordination, queued writes,
    SQL compatibility, local Web Console launch, sync/reactive inspection,
    structured error diagnostics, and Lua extension discovery are wired into the
    desktop workbench within the documented ADR boundaries
@@ -78,7 +78,7 @@ dart run tool/stage_decentdb_native.dart --bundle build/linux/x64/release/bundle
 dart run tool/stage_decentdb_native.dart --bundle build/linux/x64/release/bundle --verify-only
 ```
 
-The app expects a compatible DecentDB v2.8.0 native library to be available via:
+The app expects a compatible DecentDB v2.17.0 native library to be available via:
 
 1. System library paths (`/usr/local/lib/`, `~/.local/lib/`)
 2. Bundled with the app

apps/decent-bench/lib/app/headless_import_runner.dart

@@ -5,6 +5,8 @@ import 'dart:io';
 import 'package:decentdb/decentdb.dart';
 import 'package:path/path.dart' as p;
 
+import '../features/workspace/infrastructure/decentdb_migration_service.dart';
+
 import '../features/import/application/import_manager.dart';
 import '../features/import/domain/import_models.dart';
 import '../features/import/infrastructure/import_execution_service.dart';
@@ -688,7 +690,23 @@ Future<HeadlessImportCliReport> _buildImportReport({
   required NativeLibraryResolver libraryResolver,
 }) async {
   final libraryPath = await libraryResolver.resolve();
-  final database = Database.open(targetPath, libraryPath: libraryPath);
+  final Database database;
+  try {
+    database = Database.open(targetPath, libraryPath: libraryPath);
+  } catch (error) {
+    if (DecentDbMigrationService.isUnsupportedFormatVersionMessage(
+      error.toString(),
+    )) {
+      throw StateError(
+        'Could not open $targetPath: this file uses a legacy DecentDB '
+        'on-disk format. Run the official decentdb-migrate tool to upgrade '
+        'it to the current format, then re-run the import. Example:\n'
+        '  decentdb-migrate --source "$targetPath" --dest '
+        '"$targetPath.upgraded.ddb"',
+      );
+    }
+    rethrow;
+  }
   try {
     final tables = database.schema.listTablesInfo()
       ..sort((left, right) => left.name.compareTo(right.name));

apps/decent-bench/lib/app/headless_quality_runner.dart

@@ -7,6 +7,7 @@ import '../features/workspace/infrastructure/data_quality_report_writer.dart';
 import '../features/workspace/infrastructure/data_quality_repository.dart';
 import '../features/workspace/infrastructure/data_quality_runner.dart';
 import '../features/workspace/infrastructure/decentdb_bridge.dart';
+import '../features/workspace/infrastructure/decentdb_migration_service.dart';
 import '../features/workspace/infrastructure/native_library_resolver.dart';
 import 'startup_launch_options.dart';
 
@@ -118,6 +119,16 @@ Future<int> runHeadlessQualityCli(
       await gateway.openDatabase(databasePath);
     } catch (error) {
       writeStderr('Could not open database: $error');
+      if (DecentDbMigrationService.isUnsupportedFormatVersionMessage(
+        error.toString(),
+      )) {
+        writeStderr(
+          'This file uses a legacy DecentDB on-disk format. Run the official '
+          'decentdb-migrate tool to upgrade it in place, then re-run this '
+          'command. For example: decentdb-migrate --source <db> '
+          '--dest <db>.upgraded.ddb',
+        );
+      }
       return 3;
     }