v2.15.0 - #56
Merged
Merged
Conversation
- Introduced named database profiles: `default`, `balanced`, `low_memory`, `embedded_fast`, and `tuned_durable`. - Implemented `db_config_profile` function to parse and apply profiles from options. - Updated `db_config_from_options` to handle profile options and override with explicit settings. - Enhanced documentation to reflect new profile options and usage examples in C, .NET, and configuration guides. - Added tests for profile parsing and validation of unknown profiles. - Created a detailed performance issues document outlining gaps versus SQLite and a plan to address them.
- Introduced a new markdown file `2026-06-20-PERF_ISSUES_PROMPT.md` outlining the strategy to close performance gaps between DecentDB and SQLite, detailing the problem statement, constraints, primary files, and a phased improvement plan. - Added a new Python script `benchmark_runner.py` to automate the benchmarking process, including building the DecentDB library, running benchmarks, and summarizing results using the Rich library for enhanced output visualization.
…ions - Introduced a new native decoder `decode_matrix_i64_text_f64_i64_i64` in `_fastdecode.c` to optimize decoding for a specific 5-column structure. - Updated `__init__.py` to integrate the new decoder into the Python bindings, allowing for efficient handling of SQL queries with 5-column results. - Modified `mod.rs` to capture residual predicates on non-range columns during filtered projections, improving query performance by allowing more complex conditions. - Implemented logic to evaluate residual predicates directly against stored row values, bypassing the generic executor for better efficiency. - Added tests to validate the functionality of residual predicates in SQL queries, ensuring correctness and performance improvements. - Documented performance benchmarks showing significant speedups for filtered range queries and indexed range/order operations compared to SQLite.
…and order support
…aching for improved DML performance
- Implemented tests for general grouped queries with qualified ORDER BY clauses to ensure correct projection values are used. - Added tests for grouped Common Table Expressions (CTEs) to validate the handling of qualified projected columns. - Introduced a test for indexed inner joins that counts distinct child values, ensuring accurate aggregation of roles and films. - Added a test for three-table genre popularity aggregates that utilizes bridge indexes for efficient querying. - Implemented a fast path for the showdown directors CTE to optimize aggregate calculations without materializing intermediate results.
…ext scoring - Introduced multiple tests for movie-related queries including indexed projections, tag searches, watchlist queries, and top-rated movies, ensuring efficient execution paths and correct results. - Enhanced the full-text indexing logic by simplifying the scoring term mapping process, improving readability and performance. - Updated sorting mechanism in full-text search results to use `sort_by_key` for clarity and efficiency.
…xing - Implemented casting from TEXT to UUID in the `cast_value` function. - Enhanced `prepare_simple_insert` to accept UUIDs as text with `CAST($n AS UUID)` in prepared statements. - Introduced typed UUID runtime B-tree keys for efficient indexing and retrieval. - Updated tests to cover UUID insertion and casting scenarios. - Improved performance of bulk inserts involving UUIDs by optimizing the prepared insert path. - Added detailed benchmarking results reflecting the performance improvements with UUID handling.
…optimize SQL operations - Refactored BM25 scoring to introduce `bm25_score_iter` for better flexibility with term inputs. - Added tests to ensure `bm25_score_iter` matches the behavior of the original `bm25_score` function. - Implemented `queue_delete_documents` in `TrigramIndex` to handle batch deletions efficiently. - Added tests for batch document deletion to verify that pending postings are correctly removed. - Enhanced full-text search tests to ensure that limiting results retains the correct top-ranking document. - Introduced a unique secondary index handling in upsert operations to ensure correct behavior on conflicts. - Added tests for upsert operations with unique constraints to validate expected outcomes. - Implemented a union operation test to verify that range projections match expected results. - Added tests for prepared statements with UUID parameters to ensure correct behavior in transactions. - Enhanced window function tests to cover additional scenarios, including rolling averages. - Updated performance notes in the design document to reflect recent optimizations and benchmarks. - Added a new C API function for binding text parameters in prepared statements to improve performance.
- Introduced a new JSON file `bench_complex_results.json` containing detailed benchmark results comparing `decentdb` and `sqlite` across various metrics. - Included performance metrics such as aggregate response times, insert rates, and query execution times for both databases. - Captured equivalence checks and detailed results for specific queries related to movie data. - Configurations for both databases and engine versions are documented within the JSON structure.
- Introduced ADR 0196 for Persisted DML and Cascade Delete Performance Path, outlining strategies to close benchmark gaps related to DML operations and cascade deletes. - Added ADR 0197 for Fulltext Runtime Index Delta Overlays, proposing a new structure to optimize fulltext index updates and reduce overhead during mutations. - Implemented ADR 0198 for Vectorized RETURNING DML Execution, focusing on transaction-local execution and prepared-plan caching to enhance performance for RETURNING statements. - Established ADR 0199 for Transaction-Local Cascade Delete Batching, detailing a new approach to batch cascade deletes while maintaining visibility and performance. - Updated the ADR README to reflect the new additions and their significance in improving DecentDB's performance against SQLite benchmarks.
…date format version to 14 - Added `encoded_prefix_matches` method to `Row` for prefix matching of encoded rows. - Introduced `crc32c_patch_bytes` function in `checksum.rs` for efficient CRC32C updates during byte patching. - Updated `DB_FORMAT_VERSION` to 14 in `header.rs` to reflect changes in the on-disk format. - Created `DELETE_BATCH.md` design document detailing performance improvements for cascade deletes. - Established ADR 0200 for the introduction of resident table delete tombstones, including migration paths and format changes. - Enhanced tests for `encoded_prefix_matches` and `crc32c_patch_bytes` to ensure correctness and performance.
- Introduced logical tombstones in `TableData` to manage deleted rows efficiently. - Enhanced `row_ids_in_range` and `row_by_id` methods to respect tombstone state. - Added `mark_existing_row_set_deleted` method to track deleted row IDs. - Updated full-text index to handle document deletions and maintain accurate search results. - Improved performance for bulk deletes and cascade deletes in resident storage. - Added tests to verify tombstone behavior and search functionality after deletions.
…ccurate SQLite comparisons
…struction and optimized token handling
- Implemented `decode_float64_at` method to read FLOAT64 values from encoded row data. - Added tests to verify correct decoding of FLOAT64 fields and handling of NULL values. fix: update WAL page reading method - Changed `pager.read_page` to `pager.read_page_from_disk` for improved page retrieval. docs: update changelog with recent enhancements and fixes - Documented performance improvements in full-text search and query metrics. - Noted fixes related to snapshot reads and wide-row column preservation.
…formance tuning - Updated comparisons between DecentDB and SQLite, clarifying version specifics and design priorities. - Expanded performance tuning section with detailed profiles, durability settings, and caching strategies. - Improved explanations of Lua extensions, including manifest requirements and permission constraints. - Added new SQL functions and examples to the SQL reference, enhancing usability for date and time operations. - Clarified transaction behavior, emphasizing durability guarantees and bulk load options. - Enhanced indexing strategies and query optimization tips for better performance insights.
- Introduced a comprehensive specification for structured error handling and diagnostics in DecentDB. - Defined a new public diagnostic contract with layers for numeric categories, diagnostic subcodes, and human-readable messages. - Established a JSON shape for error responses, including required and optional fields for detailed diagnostics. - Outlined product goals, non-goals, and implementation phases for the new diagnostic system. - Created an initial subcode catalog covering various error categories and conditions. - Implemented redaction rules to ensure sensitive information is not exposed in diagnostics. - Specified binding projection requirements for various programming languages and platforms. - Included testing requirements and compatibility rules to maintain stability across versions.
This commit introduces a new markdown file containing a comprehensive list of suggestions for future enhancements to DecentDB. The suggestions are categorized by impact and priority, including high-impact features like Online Non-Blocking DDL and Online Backup, as well as medium and lower-priority additions. The document aims to aggregate feedback from multiple coding agents to strengthen the roadmap and align with DecentDB's identity as a durable, fast, and embeddable SQL engine.
- Deleted the outdated METRIC_IMPROVEMENTS_PLAN.md file as it is no longer relevant. - Archived the previous metric improvements plan to _archive/METRIC_IMPROVEMENTS_PLAN.md for historical reference. - Updated WIN_PERFORMANCE_IMPROVEMENTS_01.md to reflect the new state of metrics, emphasizing the performance wins of DecentDB over SQLite. - Adjusted references to the archived metric improvements plan in related documents.
…ric improvement tracker
…cal backlog items including core read/query performance and Postgres-backed sync.
…o Database - Introduced a comprehensive guide comparing DecentDB and LiteDB, detailing their differences in data models, use cases, and features. - Added a comparison document for DecentDB and PGlite, highlighting their strengths in local SQL workloads and ecosystem compatibility. - Created a detailed comparison between DecentDB and Turso Database, focusing on their unique features, durability, and SQL capabilities.
… signals for libSQL, PGlite, and others
- Implemented caching of parsed view queries in the runtime, seeded by `CREATE VIEW`. - Optimized projected deferred reads for view filter and limit paths, reducing unnecessary column decoding. - Introduced streaming deferred view joins to avoid materializing intermediate row vectors. - Trimmed fetched join columns from projected deferred rows based on proven indexes. - Lowered the threshold for deferred-view limit fast-path to improve performance for small-scale queries. - Added bounded top-N postprocessing for efficient handling of simple projection with ORDER BY and LIMIT. - Updated planner `EXPLAIN` output to include metadata on expanded-view filter, projection, and limit pushdown opportunities. - Reduced overhead in the faulty VFS wrapper for passive paths, maintaining failpoint behavior while improving performance.
- Trimmed row-id alias join keys from deferred SQL view projected reads, allowing for more efficient three-table inner-join view chains by synthesizing join keys directly from stored row ids. - Fixed a planner index-detection gap, enabling cost-based `IndexedJoin` selection for explicit `JOIN ... ON` syntax when a useful B-tree index exists, preventing unnecessary fallbacks to `HashJoin`. - Enhanced `EXPLAIN` to surface the chosen `IndexedJoin` operator along with estimates for explicit inner equi-joins. - Added regression tests for planner `EXPLAIN` to validate indexed-join selection, hash-join selection without a useful index, and the surfacing of estimated rows and costs. test: add tests for join optimizations and explain output - Introduced tests to verify the selection of indexed joins when useful indexes are available and to ensure correct behavior when no useful index exists. - Added tests to confirm that `EXPLAIN` outputs the expected estimates for join operations.
- Finalize the WIN01 read/query performance closeout, ensuring public `embedded_compare` guardrails pass for balanced, low-memory, and tuned durable DecentDB profiles. - Achieve over 25% reduction in rust-baseline full/huge peak RSS and exceed 2x improvement in full/huge view-query gates without compromising WAL sync or durability semantics. - Enhance public prepared-insert and read hot paths with transaction-local next-row-id caching, reduced redundant catalog writes, and optimized result construction. - Update the Future Wins roadmap to reflect WIN01 as delivered, prioritizing cross-binding cursor/row-view parity as the next performance-adjacent item.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request includes a version bump across the Rust workspace, Dart bindings, and Flutter integration to
2.15.0, and updates benchmark documentation to reference a new performance plan. It also adds new benchmark result files for several dataset scales and introduces a new C API function in the Dart native header.Version Updates:
2.15.0inCargo.toml, all Dart/Flutterpubspec.yamlfiles, Flutterbuild.gradle,decentdb_flutter.podspec, and theVERSIONfile. [1] [2] [3] [4] [5] [6] [7]Benchmark Documentation and Results:
benchmarks/rust-baseline/README.mdto referenceWIN_PERFORMANCE_IMPROVEMENTS_01.mdinstead of the previous metric plan. [1] [2]smoke,medium,full, andhugedataset scales underbenchmarks/rust-baseline/results/. [1] [2] [3] [4]Dart Native API:
ddb_stmt_bind_text_step_row_viewtobindings/dart/native/decentdb.hfor stepping through query results with a bound text parameter.