Skip to content

Add sqlite-vec extension#7

Merged
ocavue merged 6 commits intomasterfrom
sqlite-vec-simd
Apr 22, 2025
Merged

Add sqlite-vec extension#7
ocavue merged 6 commits intomasterfrom
sqlite-vec-simd

Conversation

@ocavue
Copy link

@ocavue ocavue commented Apr 22, 2025

Checklist

  • I grant to recipients of this Project distribution a perpetual,
    non-exclusive, royalty-free, irrevocable copyright license to reproduce, prepare
    derivative works of, publicly display, sublicense, and distribute this
    Contribution and such derivative works.
  • I certify that I am legally entitled to grant this license, and that this
    Contribution contains no content requiring a license from any third party.

Summary by CodeRabbit

  • New Features

    • Integrated the sqlite-vec extension with SIMD support, enabling enhanced vector operations in SQLite.
    • The extension is now automatically loaded and initialized when the application starts.
  • Chores

    • Updated build process to include and compile the sqlite-vec extension for both debug and distribution builds.

@coderabbitai
Copy link

coderabbitai bot commented Apr 22, 2025

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (5)
  • dist/wa-sqlite-async.mjs is excluded by !**/dist/**
  • dist/wa-sqlite-async.wasm is excluded by !**/dist/**, !**/*.wasm
  • dist/wa-sqlite-jspi.wasm is excluded by !**/dist/**, !**/*.wasm
  • dist/wa-sqlite.mjs is excluded by !**/dist/**
  • dist/wa-sqlite.wasm is excluded by !**/dist/**, !**/*.wasm

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The changes introduce integration of the sqlite-vec extension (version 0.1.6) into the project build and runtime. The Makefile is updated to download, extract, and compile the sqlite-vec source with SIMD support, and to include it in both debug and distribution builds. In the application source, the sqlite-vec extension is registered as an automatic SQLite extension, ensuring it is initialized whenever SQLite is used. No changes are made to public APIs, and the updates are confined to build configuration and application initialization.

Changes

File(s) Change Summary
Makefile Added integration for sqlite-vec v0.1.6: variables, download/extract rules, SIMD flags, source inclusion, and build targets.
src/main.c Declared sqlite3_vec_init and registered it as an auto-extension after SQLite initialization.

Sequence Diagram(s)

sequenceDiagram
    participant MainApp
    participant SQLite
    participant sqlite3_vec_init

    MainApp->>SQLite: sqlite3_initialize()
    MainApp->>SQLite: sqlite3_auto_extension(&sqlite3_vec_init)
    Note right of SQLite: Registers sqlite3_vec_init as an auto-extension
    SQLite->>sqlite3_vec_init: (On database open) Initialize extension
    sqlite3_vec_init-->>SQLite: Extension ready
Loading

Possibly related PRs

  • Add better-trigram extension. #6: Adds integration and automatic loading of the sqlite-bettertrigram extension, similar in structure and purpose but targeting a different SQLite extension.

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
Makefile (1)

9-14: Consider adding SHA verification for sqlite-vec

While the extension-functions.c download has SHA3 verification (lines 157-162), there's no similar verification for the sqlite-vec download. Consider adding a hash verification step to ensure the integrity of the downloaded sqlite-vec package.

src/main.c (1)

18-18: Proper registration of sqlite-vec extension

The sqlite-vec extension is correctly registered as an automatic SQLite extension.

However, consider adding error handling for the sqlite3_auto_extension call, which returns an error code:

-  sqlite3_auto_extension(&sqlite3_vec_init);
+  int rc = sqlite3_auto_extension(&sqlite3_vec_init);
+  if (rc != SQLITE_OK) {
+    // Log or handle the error
+  }
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c589eec and f798220.

⛔ Files ignored due to path filters (6)
  • dist/wa-sqlite-async.mjs is excluded by !**/dist/**
  • dist/wa-sqlite-async.wasm is excluded by !**/dist/**, !**/*.wasm
  • dist/wa-sqlite-jspi.mjs is excluded by !**/dist/**
  • dist/wa-sqlite-jspi.wasm is excluded by !**/dist/**, !**/*.wasm
  • dist/wa-sqlite.mjs is excluded by !**/dist/**
  • dist/wa-sqlite.wasm is excluded by !**/dist/**, !**/*.wasm
📒 Files selected for processing (2)
  • Makefile (5 hunks)
  • src/main.c (2 hunks)
🔇 Additional comments (8)
Makefile (6)

9-14: Setup for sqlite-vec extension looks good

The variables for the sqlite-vec extension are properly defined with clear naming conventions that match the project's style.


25-26: Proper inclusion of extension files

Extension files are correctly added to the compilation list.


39-39: Appropriate vpath configuration

Adding the sqlite-vec directory to vpath ensures the build system can locate the extension files correctly.


56-56: SIMD support correctly enabled

The SIMD-specific compiler flags -DSQLITE_VEC_ENABLE_WASM_SIMD -msimd128 are correctly added to enable SIMD support for the extension.


181-187: Separate compilation rules for sqlite-vec

The separate compilation rules for debug and distribution builds of sqlite-vec.c are appropriately defined, ensuring that the extension is compiled with the correct flags.


164-167:

Details

✅ Verification successful

Dependency rule for sqlite-vec

The rule to download and extract the sqlite-vec extension source is properly defined.

However, consider verifying that the URL consistently points to the intended version:


🏁 Script executed:

#!/bin/bash
# Verify that the URL is accessible and contains the expected sqlite-vec version
curl -sI ${EXTENSION_VEC_URL} | grep "HTTP/"

Length of output: 111


🏁 Script executed:

#!/bin/bash
# Extract the value of EXTENSION_VEC_URL from the Makefile and verify it returns a valid HTTP response
url=$(grep -E '^EXTENSION_VEC_URL\s*=' Makefile | sed 's/^[^=]*=\s*//')
echo "Checking URL: $url"
curl -sI "$url" | grep HTTP/

Length of output: 723


🏁 Script executed:

#!/bin/bash
# Extract version and URL template from the Makefile
version=$(grep -E '^EXTENSION_VEC_VERSION\s*=' Makefile | sed 's/^[^=]*=[[:space:]]*//')
url_template=$(grep -E '^EXTENSION_VEC_URL\s*=' Makefile | sed 's/^[^=]*=[[:space:]]*//')

# Substitute the version placeholder
url=${url_template//\$\{EXTENSION_VEC_VERSION\}/$version}

echo "Checking resolved URL: $url"
curl -sI "$url" | grep HTTP/

Length of output: 852


Dependency download URL verified

The Makefile rule for downloading sqlite-vec correctly interpolates EXTENSION_VEC_VERSION (0.1.6) and points to a valid GitHub release asset. A curl -sI check on the resolved URL returned HTTP/2 302, confirming the file exists and is served. No further changes needed.

src/main.c (2)

5-6: Correct external function declaration

The external declaration for sqlite3_vec_init is properly defined.


5-18:

Details

✅ Verification successful

Verify SIMD compatibility with target environments

The changes enable SIMD support in the WebAssembly build, which may not be supported in all browsers. Please verify that this is compatible with your target environments.


🌐 Web query:

Which browsers support WebAssembly SIMD?

💡 Result:

WebAssembly SIMD Browser Support

WebAssembly SIMD (Single Instruction, Multiple Data) enables parallel processing for improved performance in web applications, especially those using compute-heavy algorithms. As of April 2025, support for WebAssembly SIMD is extensive across major browsers.

Desktop Browsers

Browser Supported Versions Notes
Chrome 91 and above Full support since May 2021
Firefox 89 and above Full support since June 2021
Edge 91 and above Shares engine with Chrome; full support from version 91
Safari 16.4 and above Full support since March 2023
Opera 77 and above Full support
  • Internet Explorer: Not supported[4].

Mobile Browsers

Browser Supported Versions Notes
Safari on iOS 16.6 and above Full support[4]
Chrome for Android All modern versions Full support[4]
Firefox for Android All modern versions Full support[4]
Samsung Internet 16 and above Full support[4]
Opera Mobile 80 Full support[4]

Additional Notes

  • Node.js: SIMD is supported in Node.js version 16.4 and up[7].
  • Security Modes: Some enhanced security modes in browsers like Edge or iOS Lockdown mode may disable SIMD even if it is technically supported, so real-world availability can depend on user or enterprise security settings[6].
  • Feature Detection: It’s recommended to use runtime feature detection (such as the wasm-feature-detect library) to ensure SIMD availability before relying on it, since users may have incompatible configurations despite using a supported browser version[2][5].

Summary

All major modern browsers (Chrome, Firefox, Edge, Safari, Opera, and their mobile counterparts) support WebAssembly SIMD by default in their recent versions, making it safe to leverage SIMD features for performance-critical web applications, provided you check for support at runtime[3][4][7].

Citations:


WebAssembly SIMD compatibility verified
All major modern browsers (Chrome 91+, Firefox 89+, Edge 91+, Safari 16.4+, Opera 77+) and their mobile counterparts, as well as Node.js 16.4+, support WebAssembly SIMD by default. No code changes are required to enable SIMD in these environments.

• If you need broader coverage or graceful degradation in rare security‑restricted modes, add a runtime check (for example, via the wasm-feature-detect library) before invoking SIMD‑optimized code.

@ocavue ocavue changed the title Sqlite vec simd Add sqlite-vec extension Apr 22, 2025
@ocavue ocavue merged commit 1ffca81 into master Apr 22, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants