Repository navigation
fix: run the local embedding model on onnxruntime-web - #526
Merged
Merged
Conversation
Installing 2.0.0 fails on Linux x64 without access to nuget.org: @huggingface/transformers depends on onnxruntime-node, whose install script downloads CUDA binaries and fails the install when that download fails. A dependent package cannot turn the download off. Run all-MiniLM-L6-v2 on onnxruntime-web (WebAssembly) with @huggingface/tokenizers instead. Nothing native is installed for the model, so the install works offline from nuget and on any libc; transformers, onnxruntime-node and sharp are gone. Vectors change, so existing vector indexes need `libscope admin reindex --rebuild`: - Full-precision model.onnx instead of model_quantized.onnx. The quantized model's vectors drift from the original model's (cosine min 0.978 on 100 indexed chunks); the WebAssembly runtime reproduces the full-precision model exactly (cosine 1.00000 against Python onnxruntime). - Truncation at 256 tokens keeping [SEP], as sentence-transformers (max_seq_length 256) does. - The local provider's model id is now sentence-transformers/all-MiniLM-L6-v2, so 2.0.0 indexes no longer match and ask for the rebuild. Migration 20 records a placeholder model for 1.x indexes, which recorded none. The model is pinned to a Hugging Face revision and downloaded with progress into ~/.libscope/models (about 90 MB). Inference uses one WebAssembly thread per core; it is about 2.5-3x slower than the native quantized model was.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
npm install -g libscope@2.0.0fails on Linux x64 when nuget.org is unreachable.@huggingface/transformersdepends ononnxruntime-node, whose install script downloads CUDA binaries (about 270 MB) and fails the whole install when that download fails. libscope cannot turn that download off for its users:.npmrcandoverridesfrom a dependency are ignored fornpm i -gandnpx.Change
onnxruntime-web(WebAssembly) with@huggingface/tokenizers: tokenize, truncate, run, mean-pool, normalize.@huggingface/transformers,onnxruntime-nodeandsharpare removed. No native code is installed for the model.model.onnxinstead ofmodel_quantized.onnx. The model files are pinned to Hugging Face revision751bff3and cached in~/.libscope/models(about 90 MB).[SEP], as sentence-transformers does (max_seq_length: 256).min(4, cores / 2)).sentence-transformers/all-MiniLM-L6-v2, so 2.0.0 vector indexes fail the existing identity check with "Runlibscope admin reindex --rebuild". Migration 20 records a placeholder model for 1.x vector indexes, which recorded none, so they fail the same check.docs/migration-v2.md, model size and runtime updates, and the Dockerfile comment about the CUDA download is removed.Quality
Measured on 100 chunks from a real index, against Python
onnxruntimewith the Hugging Face tokenizer:tests/unit/local-embedding.test.tschecks the provider against reference vectors from Python for a short and a truncated long input. The neural retrieval-quality suite passes.Cost
Verification
npm run lint(0 errors),format:check,typecheck,test:coverage: 2138 tests pass.onnxruntime-nodeis not installed.searchstopped with the rebuild message,libscope admin reindex --rebuildre-embedded 13,275/13,275 chunks, and search returned the expected pages.Release-please should make this 2.0.1.