A cache bounded by measured bytes, that refuses a load it cannot afford instead of evicting everything and dying anyway.
A production site froze one evening. Workers died and respawned in a loop — seventy-one deaths before it was brought under control. It came back, then fell over again an hour later. No kernel OOM, no ban, nothing to do with the day's deploy.
The cause was eleven lines of ordinary code: a module-level dictionary, populated per worker, holding parsed seasons of event data. No eviction. It had been written with a comment estimating roughly 200 MB per season and the reassuring note that production would have more RAM.
The measurements, once taken:
| One season on disk | 112 MB |
| Same season, parsed, in RAM | ~320 MB |
| Five seasons × two phases × 4 workers | ~12 GB |
| Actual machine | 7.7 GB |
Any user browsing back through the archive walked the cache straight through the ceiling. The fix that evening capped the number of entries and dropped a redundant field. It held. But it was still guessing.
The number that changed how I think came later, from measuring properly rather than estimating:
one parsed game object costs 3.54 MB of real RAM, and 302 KB when pickled — the serialised size understates the truth by a factor of about twelve.
Which means the original estimate was not slightly wrong. It was wrong by an order of magnitude, in the direction that kills a server.
Caches are almost always bounded by the wrong unit. lru_cache(maxsize=128), a cap of eight entries, a dict trimmed at a thousand keys — all of them bound cardinality. Memory pressure is about bytes. Those two are only related if your entries are uniform, and in any interesting system they are not: one entry is a short season, the next is a full archive.
Serialised size is not memory size. Pickle length, len(json.dumps(...)), file size on disk — all of them are the tempting proxy, because they are easy, and all of them are wrong by a large multiple for anything made of many small Python objects. Every object header, every dict, every interned-or-not string is invisible to a serialiser and very visible to the allocator.
Eviction is the wrong answer to an entry that is simply too big. This is the part most implementations get wrong. If a single value exceeds the whole budget, evicting the rest of the cache does not help — you throw away everything useful, then run out of memory anyway. The correct response is to refuse, loudly, before you are in that state.
from bytecap import ByteCache
cache = ByteCache(budget=1_000_000_000)
season = cache.get_or_load(
"events:2026:regular",
lambda: parse_season(2026),
version=source_fingerprint,
)Four decisions do the work.
Cost is measured, not declared. The default materialiser runs the loader while watching the process's resident set, and never reports less than a deep walk of the resulting object graph. RSS deltas are noisy — the allocator may have had room, or may have grown for unrelated reasons — so the structural size acts as a floor. It is an estimate, but it is an estimate anchored to two independent measurements rather than to a guess in a comment.
Too big means refused, not evicted-for. If the measured cost exceeds the budget, TooLarge is raised and the cache is left exactly as it was. Nothing is evicted to make room for something that was never going to fit. There is a test asserting the surviving entries are untouched.
admit() lets you refuse before paying for the build. Measuring requires materialising, so the first load of an oversized value costs you the peak once. When you can estimate the cost up front — file size, row count, a partition manifest — admit(key, estimated) raises before the twenty-second build runs at all.
A version change evicts before it rebuilds. Reloading a key whose version has moved drops the old entry first, then builds. The obvious implementation builds the new value, then replaces the old one — which means the peak holds two copies of the largest thing in your system. Halving that peak is free; it just requires doing the steps in the right order. There is a test that inspects the cache's ledger from inside the loader to prove the old entry is already gone.
bytecap is a bounded box, not a memory manager and not a framework.
request ──▶ cache.get_or_load(key, loader, version)
│
├─ hit, same version ──────────▶ value
├─ hit, stale version ─▶ evict ─▶ materialise ─▶ measure ─┐
└─ miss ───────────────────────▶ materialise ─▶ measure ─┤
│
cost > budget ──▶ TooLarge │
cost ≤ budget ──▶ store, evict LRU ▶ value
It knows nothing about what it holds. It never serialises, never touches disk, never spawns anything. It is one dictionary, one lock and one integer ledger — the point is not the data structure, it is that the ledger is denominated in the unit that actually runs out.
In a multi-worker deployment, remember the budget is per process. Four workers with a 1 GB budget each is a 4 GB commitment, and that arithmetic is precisely what went wrong in the incident above.
pip install bytecap
Python 3.11+, no dependencies. Resident-set measurement uses /proc on Linux and degrades to structural size elsewhere.
Basic:
cache = ByteCache(budget=512 * 1024 * 1024)
value = cache.get_or_load("key", build_it)Refusing before an expensive build:
from bytecap import TooLarge
try:
cache.admit(key, estimated=partition.bytes_on_disk * 3)
except TooLarge:
return stream_from_disk(key)
value = cache.get_or_load(key, lambda: load(partition))Invalidating by version rather than by hand:
cache.get_or_load("events:2026", loader, version=fingerprint_of_sources())Deterministic measurement, for tests or for a machine without /proc:
from bytecap import ByteCache, measure_deep_size
cache = ByteCache(budget=10_000_000, materialize=measure_deep_size)Watching it:
stats = cache.stats()
log.info("%s: %.0f%% of budget, %d refusals", cache.name, stats.fill * 100, stats.refusals)No stampede protection. The lock is released while the loader runs, so two threads asking for the same missing key will both build it. Holding the lock across an arbitrary user callback is a far worse failure — one slow load would block every reader in the process. The ledger is protected against the resulting race, and there is a test hammering it from four threads.
No TTL. Time is not a good proxy for correctness. version= covers the real case, which is "the inputs moved", and it does so exactly rather than eventually. If you need a clock, put it in the version.
No async variant. The lock is a threading.RLock and the loader is synchronous. An async cache is a different object with different failure modes, and pretending one class can be both is how you get a footgun.
RSS is not exact, and the README says so rather than hiding it. A precise per-object accounting in CPython does not exist. Two crude measurements combined with a max() is not elegant, but it is honest, it is an order of magnitude closer than a serialiser, and it is injectable when you need determinism.
pip install -e ".[dev]"
pytest
The ones that exist because something went wrong:
| Test | Failure it pins down |
|---|---|
test_reload_of_changed_version_evicts_before_building |
two copies of the largest object at peak |
test_entry_larger_than_budget_is_refused |
the OOM the incident actually was |
test_refused_entry_does_not_evict_the_rest_of_the_cache |
throwing away the cache and dying anyway |
test_admit_refuses_before_the_expensive_build_runs |
paying for a build whose result cannot be kept |
test_concurrent_readers_do_not_corrupt_the_ledger |
double-counted bytes when two threads race on one key |
MIT.