Skip to content

Add VFSFind so a VFS can delegate temp files to the base VFS - #20

Open
jtarchie wants to merge 1 commit into
psanford:mainfrom
jtarchie:divert-temp-files
Open

Add VFSFind so a VFS can delegate temp files to the base VFS#20
jtarchie wants to merge 1 commit into
psanford:mainfrom
jtarchie:divert-temp-files

Conversation

@jtarchie

@jtarchie jtarchie commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Closes #15.

Problem

When a query is too big to sort in memory, SQLite opens a temporary file through the same VFS that serves the database. A read-only VFS (compressed, remote, etc.) can't create files, so the open fails and the whole query fails with unable to open database file. The only workaround was making every caller run PRAGMA temp_store = memory.

Fix

Adds VFSFind(name string) VFS (the API sketched in #15). It looks up a VFS SQLite already has registered — pass "" for the default one — and returns it as a VFS that Go code can call. A read-only VFS can then forward temp files (SQLite asks for them with an empty file name) to a VFS that can create them, the same way SQLite's own appendvfs does:

func (vfs *MyVFS) Open(name string, flags sqlite3vfs.OpenFlag) (sqlite3vfs.File, sqlite3vfs.OpenFlag, error) {
	if name == "" { // temp file
		return vfs.base.Open(name, flags)
	}
	// ... open the main database yourself
}

Test

TestDelegateTempFiles runs a query that must spill, against a VFS that can't make temp files. Without a base VFS it fails (the bug from #15); delegating to VFSFind("") it passes.

jtarchie added a commit to jtarchie/sqlitezstd that referenced this pull request Sep 2, 2026
Only the main database is a zstd archive. Everything else sqlite asks
this VFS for -- above all the temp file a sorter or a transient index
spills into -- it cannot create, and until now the open failed and took
the query with it: "unable to open database file". The documented
workaround was to require PRAGMA temp_store = memory of every caller,
which a library cannot enforce and which callers who do not spill never
learn they needed.

sqlite's own appendvfs diverts anything that is not the main database to
the VFS underneath it. VFSFind (psanford/sqlite3vfs#20) makes that
reachable from here, so Open does the same.

The new spec sorts a million rows through a temp file with
temp_store = FILE, and fails with the old "unable to open database file"
without the diversion.

The replace directive comes out once the upstream PR lands.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMBg7JBvZJRRu1RytUj5c3
A VFS that cannot create files has nowhere to put the temp file sqlite
opens when a sorter or a transient index spills, and there is no way to
reach the VFS underneath it to ask. The open fails, and with it the
query: "unable to open database file". The workaround is to require
`PRAGMA temp_store = memory` of every caller, which a library cannot
enforce.

sqlite's own appendvfs diverts anything that is not the main database to
ORIGVFS(pVfs)->xOpen. VFSFind makes the same thing possible here: it
wraps sqlite3_vfs_find and returns the registered VFS as a VFS this
package can call, so an implementation can pass on the files it cannot
serve.

The C side wraps the sqlite3_vfs and sqlite3_io_methods function
pointers; the Go side is a VFS/File pair over them. The sqlite3_file is
allocated with sqlite3_malloc, since sqlite holds it for the life of the
file and Go memory may not be used for that.

ReadAt reports a short read as (len(p), io.EOF). xRead does not say how
many bytes it read, only that it zero-filled the rest, and the bytes it
did read are in the buffer -- returning a smaller n would get them
overwritten with zeros on the way back to sqlite.

Fixes psanford#15

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMBg7JBvZJRRu1RytUj5c3
jtarchie added a commit to jtarchie/sqlitezstd that referenced this pull request Sep 2, 2026
Same two fixes (psanford/sqlite3vfs#20 and #21); the branches were
amended to carry fewer comments, so the merge commit moved.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMBg7JBvZJRRu1RytUj5c3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

temporary files cannot diverted

1 participant