Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ updates:
groups:
minor-and-patch:
update-types: ["minor", "patch"]
ignore:
# undici 8 requires Node.js >= 22.19; libscope supports Node.js 20.
- dependency-name: "undici"
update-types: ["version-update:semver-major"]
- package-ecosystem: "pip"
directory: "/sdk/python"
target-branch: "development"
Expand Down
18 changes: 10 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
lint-and-typecheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
- uses: actions/setup-node@v6
with:
node-version: 22
Expand All @@ -32,15 +32,17 @@ jobs:
matrix:
node-version: [20, 22]
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
- uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node-version }}
cache: npm
- run: npm ci --ignore-scripts
# Build only the native addons the tests load: better-sqlite3, and sharp
# (@xenova/transformers loads it in the neural retrieval-quality suite).
- run: npm rebuild better-sqlite3 sharp
# Build only the native addon the tests load: better-sqlite3. sharp and
# onnxruntime-node (@huggingface/transformers, used by the neural
# retrieval-quality suite) ship prebuilt binaries; the onnxruntime-node
# install script only downloads optional CUDA libraries.
- run: npm rebuild better-sqlite3
- run: npm run test:coverage
- name: Upload coverage
if: matrix.node-version == 22
Expand All @@ -55,16 +57,16 @@ jobs:
permissions:
contents: read
steps:
- uses: actions/checkout@v6
- uses: actions/dependency-review-action@v4
- uses: actions/checkout@v7
- uses: actions/dependency-review-action@v5
with:
fail-on-severity: high

build:
runs-on: ubuntu-latest
needs: [lint-and-typecheck, test]
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
- uses: actions/setup-node@v6
with:
node-version: 22
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
matrix:
language: [javascript-typescript]
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
- name: Initialize CodeQL
uses: github/codeql-action/init@v4
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
contents: read
packages: write
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
- uses: docker/setup-buildx-action@v4
- uses: docker/login-action@v4
if: github.event_name != 'pull_request'
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release-please.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
release_created: ${{ steps.release.outputs.release_created }}
tag_name: ${{ steps.release.outputs.tag_name }}
steps:
- uses: googleapis/release-please-action@v4
- uses: googleapis/release-please-action@v5
id: release
with:
release-type: node
Expand All @@ -30,7 +30,7 @@ jobs:
contents: read
id-token: write
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
- uses: actions/setup-node@v6
with:
node-version: 22
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release-python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
runs-on: ubuntu-latest
environment: pypi
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7

- uses: actions/setup-python@v6
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/sdk-go.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
matrix:
go-version: ["1.21", "1.22"]
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
- uses: actions/setup-go@v6
with:
go-version: ${{ matrix.go-version }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/sdk-python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
matrix:
python-version: ["3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
- uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
Expand Down
4 changes: 3 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# Debian (glibc) base: the local embedding model (onnxruntime-node) does not run on Alpine (musl).
# sharp and onnxruntime-node ship prebuilt binaries, so only better-sqlite3 is rebuilt. Skipping the
# onnxruntime-node install script also skips its optional CUDA download (about 270 MB).

# Build stage
FROM node:22-bookworm-slim AS builder
Expand All @@ -19,7 +21,7 @@ COPY package*.json ./
RUN apt-get update && \
apt-get install -y --no-install-recommends g++ make python3 && \
npm ci --omit=dev --ignore-scripts && \
npm rebuild better-sqlite3 sharp && \
npm rebuild better-sqlite3 && \
npm cache clean --force && \
apt-get purge -y g++ make python3 && \
apt-get autoremove -y && \
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ src/
│ ├── embedding.ts # EmbeddingProvider interface
│ ├── index.ts # createEmbeddingProvider() from config
│ ├── dimensions.ts # vector sizes of known models
│ ├── local.ts # @xenova/transformers (all-MiniLM-L6-v2)
│ ├── local.ts # @huggingface/transformers (all-MiniLM-L6-v2)
│ ├── ollama.ts # Ollama HTTP API
│ └── openai.ts # OpenAI embeddings API
├── registry/ # git pack registries: config, git, sync, search, resolve, publish, checksum
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Embeddings turn text into vectors for semantic search. LibScope supports three p
| `ollama` | | Ollama running locally | Uses nomic-embed-text by default |
| `openai` | | API key | Uses text-embedding-3-small by default |

The local provider works out of the box — no API keys, no external services. It runs the model in-process using `@xenova/transformers`.
The local provider works out of the box — no API keys, no external services. It runs the model in-process using `@huggingface/transformers` and caches it in `~/.libscope/models`.

```bash
# Switch to Ollama
Expand Down
15 changes: 8 additions & 7 deletions docs/migration-v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,14 @@ These variables do not change: `LIBSCOPE_EMBEDDING_PROVIDER`, `LIBSCOPE_LLM_PROV

### Behavior changes

| 1.x | 2.0 |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| API keys could be stored in `config.json` or `.libscope.json`. | API keys are read only from environment variables and `~/.libscope/secrets.json` (mode `0600`). Keys in `config.json` or `.libscope.json` are ignored with a warning. Move them with `libscope config set openai.apiKey <key>` or `libscope config set anthropic.apiKey <key>`. |
| `config set` refused API keys. | `config set openai.apiKey` and `config set anthropic.apiKey` write to `~/.libscope/secrets.json`. |
| `llm.provider` had no default; `ask` failed until it was set. | `llm.provider` defaults to `auto`: passthrough under MCP, else OpenAI if an OpenAI key is set, else Anthropic if an Anthropic key is set, else Ollama if `llm.url` is set or the embedding provider is `ollama`. |
| An invalid value in a config file was used as is. | An invalid value is ignored with a warning, and the default (or lower layer) applies. |
| `LIBSCOPE_ALLOW_PRIVATE_URLS=false` did not override `true` in a file. | `LIBSCOPE_INDEXING_ALLOW_PRIVATE_URLS=false` (or `0`) sets `false`. |
| 1.x | 2.0 |
| --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| API keys could be stored in `config.json` or `.libscope.json`. | API keys are read only from environment variables and `~/.libscope/secrets.json` (mode `0600`). Keys in `config.json` or `.libscope.json` are ignored with a warning. Move them with `libscope config set openai.apiKey <key>` or `libscope config set anthropic.apiKey <key>`. |
| `config set` refused API keys. | `config set openai.apiKey` and `config set anthropic.apiKey` write to `~/.libscope/secrets.json`. |
| `llm.provider` had no default; `ask` failed until it was set. | `llm.provider` defaults to `auto`: passthrough under MCP, else OpenAI if an OpenAI key is set, else Anthropic if an Anthropic key is set, else Ollama if `llm.url` is set or the embedding provider is `ollama`. |
| An invalid value in a config file was used as is. | An invalid value is ignored with a warning, and the default (or lower layer) applies. |
| `LIBSCOPE_ALLOW_PRIVATE_URLS=false` did not override `true` in a file. | `LIBSCOPE_INDEXING_ALLOW_PRIVATE_URLS=false` (or `0`) sets `false`. |
| The local embedding model ran on `@xenova/transformers` and was cached in `node_modules/@xenova/transformers/.cache`. | The local embedding model runs on `@huggingface/transformers` and is cached in `~/.libscope/models` (`/data/.libscope/models` in the Docker image). The first command that embeds text after the upgrade downloads the model again, once. The model (all-MiniLM-L6-v2, quantized) and its 384-dimension vectors do not change, so you do not have to reindex. |

### Library (SDK) code

Expand Down
Loading
Loading