Add an in-memory storage backend

[onslaughtq]

Adds newInMemoryKiokuDB/withInMemoryKiokuDB, which hold the same content-addressed representation as an on-disk database in an in-process map instead of files, so consumers (such as test suites) can build and query databases without any filesystem access.

All object reads/writes now flow through storage primitives that dispatch on the backend. The file backend keeps its existing streaming, mmap-based behavior; the trie index writer is generalized from a Handle to a byte sink to support both. The query test suite runs every case against both backends and property-checks that they give identical results.

[Copilot]

Pull request overview

Adds an in-memory KiokuDB storage backend alongside the existing file-backed implementation, routing reads/writes through backend-specific storage primitives so callers (notably tests) can use Kioku without filesystem access.

Changes:

• Introduces Storage (FileStorage / MemoryStorage) and storage primitives (storageRead/write/list/remove, createBlob) to unify persistence behavior.
• Generalizes trie index writing to target a byte sink rather than a Handle, enabling both file and memory backends.
• Updates the query test suite to run against both backends and adds a property asserting backend result equivalence.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/QueryTest.hs	Runs query tests against both file and in-memory backends; adds an equivalence property test.
src/Database/Kioku/Internal/TrieIndex.hs	Generalizes index writing to stream to a ByteString sink instead of a Handle.
src/Database/Kioku/Internal/KiokuDB.hs	Introduces Storage abstraction and backend-dispatching read/write/list/remove plus createBlob.
src/Database/Kioku/Internal/Buffer.hs	Adds bufferFromByteString to support in-memory buffers.
src/Database/Kioku/Core.hs	Exposes newInMemoryKiokuDB / withInMemoryKiokuDB and routes dataset/index creation through createBlob.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[onslaughtq]

Good catch — the file backend tests now open each database in its own withSystemTempDirectory "kioku-test", nested outside withKiokuDB so the directory is only removed after the mmapped buffers are unmapped. temporary was already a library dependency, so it just needed adding to the test suite. Fixed in 5ce817b.

---

🤖 This reply was written by Claude Code.

[onslaughtq]

Agreed — switched to atomicModifyIORef'. Fixed in 5ce817b.

---

🤖 This reply was written by Claude Code.

[onslaughtq]

Agreed — switched to atomicModifyIORef'. Fixed in 5ce817b.

---

🤖 This reply was written by Claude Code.

[onslaughtq]

Agreed — the shared map insert now uses atomicModifyIORef'. The builderRef in the same branch is left as modifyIORef' since it is a per-call local ref whose sink is driven sequentially by writeRows/writeIndex. Fixed in 5ce817b.

---

🤖 This reply was written by Claude Code.

[onslaughtq]

Agreed — deduped via Data.Set instead. The hand-rolled local stripPrefix went away at the same time in favor of Data.List.stripPrefix, which has identical semantics here. Fixed in 5ce817b.

---

🤖 This reply was written by Claude Code.

[qxjit]

The copilot feedback is worthwhile here, but I think this look good aside from those things.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/Database/Kioku/Internal/Buffer.hs:33

• Comment grammar: “does not using” / “do not depend end” reads like a typo and is a bit hard to parse.

-- This function does not using a length header to determine where
-- to stop reading, so it can only be used with Memorizable instances
-- that do not depend end of the ByteString for parsing

src/Database/Kioku/Internal/KiokuDB.hs:154

• createBlob (file backend) does not guarantee the temp Handle is closed and the temp file is removed if writer (or hashing/rename) throws. That can leak file descriptors and leave behind tmp artifacts (and can break cleanup on platforms that lock open files). Wrap the temp file lifecycle in bracketOnError/finally so resources are always released.

    FileStorage root -> do
      (tmpFile, h) <- openTempFile (root </> tmpPath) name
      result <- writer (BS.hPutStr h)
      hClose h

      sha <- hashBytes <$> LBS.readFile tmpFile
      renameFile tmpFile (root </> dataFilePath sha)
      pure (sha, result)

[onslaughtq]

Ran a deeper review pass over the whole branch after the Copilot fixes; 14cd263 addresses what it turned up.

createBlob file branch was not exception-safe. There is no cleanup between openTempFile and hClose/renameFile, so a throwing writer orphaned both the handle and the temp file — and nothing ever cleans tmp/, since gcKiokuDB walks only data/. This PR widened the window: createIndex used to call openTempFile inside the flushIndex callback, i.e. after buildSortedOffsetArray finished, whereas createBlob opens it before invoking the writer, so the handle is now held across the entire sort (including every keyFunc call). Wrapped in onException with hClose + removePathForcibly. Confirmed with a throwaway probe: before the fix a failed createDataSet leaves tmp/probe1-0 behind, after it tmp/ is empty.

Missing objects threw different exception types per backend. storageRead raised KiokuException under MemoryStorage but let BS.readFile throw an isDoesNotExistError IOException under FileStorage. Since the point of the memory backend is to stand in for a real database, a caller handling a not-yet-created dataset/index via isDoesNotExistError would pass in memory and catch nothing in production — the probe confirmed the KiokuException escapes an IOException handler outright. The memory branch now raises the same doesNotExistErrorType IOError; KiokuException stays for corrupt index/schema content.

Also in that commit: writeRows' row counter is strict (it was building an N-deep thunk chain), and the equivalence test's comment no longer claims to prevent representation drift — it only compares query results, each produced wholly within its own backend.

Consciously left alone: storageList returns [] for a path that names nothing under MemoryStorage but throws under FileStorage. Nothing reaches it today (openKiokuDB creates the top-level directories, writeObjFile the per-namespace ones), so it is a latent trap rather than a live bug; it is now documented on the function instead of changed. Asserting cross-backend SHA equality would pin the "same content-addressed representation" claim properly, but it needs Database.Kioku.Internal.KiokuDB in exposed-modules, which did not seem worth publishing internals for.

---

🤖 This comment was written by Claude Code.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/Database/Kioku/Internal/KiokuDB.hs:170

• In createBlob (file backend), the cleanup action calls hClose h unconditionally. If an exception occurs after the explicit hClose h in the normal path (e.g., during hashing or renameFile), onException will run cleanup and hClose will likely throw “handle is closed”, masking the original exception and potentially preventing temp file cleanup.

        cleanup = do
          hClose h
          removePathForcibly tmpFile