iOS: `releaseContext` does not free Metal memory — `delete ctx` runs on a pool worker with no `@autoreleasepool`

[juliusbangert]

We've been carrying a small patch for this since June and it's held up across every llama.rn version since 0.12.4, so I think it's worth getting upstream. The short version: on iOS, releasing a context doesn't actually give the GPU memory back, and an app that releases and reloads a ~2.6 GB model (background/foreground, reloadAsync, model switching) runs out of memory on the reload.

What we saw

App with Gemma 4 E2B (~2.6 GB) on an 8 GB iPhone. Release the context on backgrounding, initLlama again on foreground. The reload fails with Failed to load model, reliably within the first few cycles; two jetsam OOM kills in the crash logs from the same pattern. Memory instruments show ~2.4 GB of Metal buffers still resident after releaseContext has resolved.

Why

releaseContext / releaseAllContexts (and the two failed-initLlama cleanup paths) run delete ctx on a ThreadPool worker. delete ctx → llama_free → ggml_backend_free → ggml_metal_buffer_free, which does [buf->buffers[i].metal release] for each buffer. The MTLBuffer objects and whatever Metal creates while tearing them down go into the current thread's autorelease pool.

The pool workers are std::threads that loop forever (ThreadPool.cpp, for (;;) { … task(); }). A thread's implicit autorelease pool only drains when the thread exits, so on those threads it never drains. The C++ call chain returns, the JS promise resolves, and the Metal memory stays allocated until process exit.

Every other Metal operation in the vendored ggml-metal wraps itself in @autoreleasepool — ggml-metal-device.m has nine of them (set/get async, graph compute, event record/wait, …); the buffer-free path is the one that doesn't. Calling from an Objective-C++ thread with a pool, as the examples do, hides it.

Fix we run

Wrap delete ctx at the five sites in RNLlamaJSI.cpp in an explicit pool on Apple platforms. One .mm file and one header, plus a podspec glob change so the .mm compiles when the JSI bindings are built from source:

// cpp/jsi/AutoreleasePool.h
#pragma once
#ifdef __APPLE__
#include <functional>
void rnllama_run_in_autorelease_pool(const std::function<void()>& fn);
#endif

// cpp/jsi/AutoreleasePool.mm
#include "AutoreleasePool.h"
#import <Foundation/Foundation.h>
#ifdef __APPLE__
void rnllama_run_in_autorelease_pool(const std::function<void()>& fn) {
    @autoreleasepool { fn(); }
}
#endif

// RNLlamaJSI.cpp, at each `delete ctx;`
#ifdef __APPLE__
    rnllama_run_in_autorelease_pool([ctx]() { delete ctx; });
#else
    delete ctx;
#endif

-    s.source_files = "ios/*.{h,m,mm}", "cpp/jsi/**/*.{h,cpp}"
+    s.source_files = "ios/*.{h,m,mm}", "cpp/jsi/**/*.{h,cpp,mm}"

With it, memory returns on release and the release/reload loop runs indefinitely. We verify it with a stress suite (release → reload the LLM, embedding model and TTS, 30 cycles) and it's caught the regression the one time the patch was accidentally dropped — the reload failed on essentially the first cycle without it.

An alternative would be to drain a pool per task inside the worker loop itself (@autoreleasepool around task() in ThreadPool.cpp, behind __APPLE__), which would cover any other Obj-C work that lands on the pool. I went for the narrow version because it's obviously safe; happy to do either.

Environment

llama.rn 0.12.4 → 0.12.9, iOS 18/19, A17–A18. main today still has no pool around the release sites, so it applies there.