Add VFSFind so a VFS can delegate temp files to the base VFS - #20
Open
jtarchie wants to merge 1 commit into
Open
Add VFSFind so a VFS can delegate temp files to the base VFS#20jtarchie wants to merge 1 commit into
jtarchie wants to merge 1 commit into
Conversation
jtarchie
added a commit
to jtarchie/sqlitezstd
that referenced
this pull request
Sep 2, 2026
Only the main database is a zstd archive. Everything else sqlite asks this VFS for -- above all the temp file a sorter or a transient index spills into -- it cannot create, and until now the open failed and took the query with it: "unable to open database file". The documented workaround was to require PRAGMA temp_store = memory of every caller, which a library cannot enforce and which callers who do not spill never learn they needed. sqlite's own appendvfs diverts anything that is not the main database to the VFS underneath it. VFSFind (psanford/sqlite3vfs#20) makes that reachable from here, so Open does the same. The new spec sorts a million rows through a temp file with temp_store = FILE, and fails with the old "unable to open database file" without the diversion. The replace directive comes out once the upstream PR lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WMBg7JBvZJRRu1RytUj5c3
A VFS that cannot create files has nowhere to put the temp file sqlite opens when a sorter or a transient index spills, and there is no way to reach the VFS underneath it to ask. The open fails, and with it the query: "unable to open database file". The workaround is to require `PRAGMA temp_store = memory` of every caller, which a library cannot enforce. sqlite's own appendvfs diverts anything that is not the main database to ORIGVFS(pVfs)->xOpen. VFSFind makes the same thing possible here: it wraps sqlite3_vfs_find and returns the registered VFS as a VFS this package can call, so an implementation can pass on the files it cannot serve. The C side wraps the sqlite3_vfs and sqlite3_io_methods function pointers; the Go side is a VFS/File pair over them. The sqlite3_file is allocated with sqlite3_malloc, since sqlite holds it for the life of the file and Go memory may not be used for that. ReadAt reports a short read as (len(p), io.EOF). xRead does not say how many bytes it read, only that it zero-filled the rest, and the bytes it did read are in the buffer -- returning a smaller n would get them overwritten with zeros on the way back to sqlite. Fixes psanford#15 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WMBg7JBvZJRRu1RytUj5c3
jtarchie
force-pushed
the
divert-temp-files
branch
from
September 2, 2026 02:57
51dbe68 to
400fd46
Compare
jtarchie
added a commit
to jtarchie/sqlitezstd
that referenced
this pull request
Sep 2, 2026
Same two fixes (psanford/sqlite3vfs#20 and #21); the branches were amended to carry fewer comments, so the merge commit moved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WMBg7JBvZJRRu1RytUj5c3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #15.
Problem
When a query is too big to sort in memory, SQLite opens a temporary file through the same VFS that serves the database. A read-only VFS (compressed, remote, etc.) can't create files, so the open fails and the whole query fails with
unable to open database file. The only workaround was making every caller runPRAGMA temp_store = memory.Fix
Adds
VFSFind(name string) VFS(the API sketched in #15). It looks up a VFS SQLite already has registered — pass""for the default one — and returns it as aVFSthat Go code can call. A read-only VFS can then forward temp files (SQLite asks for them with an empty file name) to a VFS that can create them, the same way SQLite's ownappendvfsdoes:Test
TestDelegateTempFilesruns a query that must spill, against a VFS that can't make temp files. Without a base VFS it fails (the bug from #15); delegating toVFSFind("")it passes.