Ingestion killed by 503 when tiktoken downloads BPE vocab from public Azure blob at runtime

[apjaphet]

Summary

Ingestion fails with a misleading 503 Server Error: The server is busy that looks like an AI Gateway error but is actually tiktoken trying to download the BPE vocab from a public Azure blob at runtime.

Root cause

The traceback originates in app/utils/tokens.py → tiktoken.get_encoding("cl100k_base") → openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken:

requests.exceptions.HTTPError: 503 Server Error: The server is busy.
  for url: https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken

A datacenter / private-gateway deployment should not depend on a public internet download at all. When that URL is throttled/unreachable, the exception propagates and kills the whole ingestion job.

Repro

1. Run with no pre-cached tiktoken vocab and the public blob unreachable/throttled.
2. Ingest any document; token counting raises and the source errors out.

Fix applied locally

app/utils/tokens.py now:

• Catches the load failure and falls back to a char-based token estimate (~4 chars/token) instead of crashing.
• Caches only successful loads (replacing lru_cache), so one transient 503 cannot permanently degrade the process to estimates.

Recommended follow-up (proper hardening)

Pre-cache the vocab in the image and set TIKTOKEN_CACHE_DIR so there is no runtime network call. The fallback should be a safety net, not the normal path.

Diff

diff --git a/app/utils/tokens.py b/app/utils/tokens.py
index 8c7e413..46e1608 100644
--- a/app/utils/tokens.py
+++ b/app/utils/tokens.py
@@ -6,18 +6,52 @@ identical to Claude/Gemini tokenizers but close enough for sizing decisions
 per char than English — cl100k_base handles this correctly.
 """
 
-from functools import lru_cache
+from loguru import logger
 
 import tiktoken
 
+# Rough chars-per-token ratio for the fallback estimator when the tiktoken BPE
+# vocab is unavailable. cl100k_base averages ~4 chars/token for English prose.
+_FALLBACK_CHARS_PER_TOKEN = 4
+
+
+_ENC_CACHE = {}
+
 
-@lru_cache(maxsize=1)
 def _encoding():
-    return tiktoken.get_encoding("cl100k_base")
+    # NOTE: get_encoding() will lazily DOWNLOAD the BPE vocab from a public
+    # Azure blob (openaipublic.blob.core.windows.net) if it isn't already
+    # cached. In a datacenter / private-gateway deployment that URL may be
+    # unreachable or rate-limited (seen as a 503), which must not take down
+    # ingestion. Returns None on failure so count_tokens() falls back to a
+    # char-based estimate. Proper fix: pre-cache the vocab in the image and
+    # set TIKTOKEN_CACHE_DIR so no runtime network call happens.
+    #
+    # We cache only the successful encoding — a transient download failure must
+    # NOT be memoized, or one 503 would degrade the whole process to estimates.
+    if "enc" in _ENC_CACHE:
+        return _ENC_CACHE["enc"]
+    try:
+        enc = tiktoken.get_encoding("cl100k_base")
+        _ENC_CACHE["enc"] = enc
+        return enc
+    except Exception as exc:
+        logger.warning(
+            f"tiktoken vocab unavailable ({type(exc).__name__}: {exc}); "
+            f"falling back to char-based token estimate"
+        )
+        return None
 
 
 def count_tokens(text: str) -> int:
-    """Return token count for `text`. Empty/None → 0."""
+    """Return token count for `text`. Empty/None → 0.
+
+    Uses tiktoken when available; falls back to a char-based estimate if the
+    vocab can't be loaded (e.g. offline / public blob returns 503).
+    """
     if not text:
         return 0
-    return len(_encoding().encode(text, disallowed_special=()))
+    enc = _encoding()
+    if enc is None:
+        return max(1, len(text) // _FALLBACK_CHARS_PER_TOKEN)
+    return len(enc.encode(text, disallowed_special=()))