help request: My APISIX is experiencing 100% CPU utilization and has become unresponsive

[zhaoqiang1980]

Description

When the free_size of Prometheus in APISIX's share_dict becomes 0, will it trigger a 100% CPU exception? This issue occurred in 2 out of 38 instances. Even after removing the traffic, the CPU remains fully occupied. This is what was observed using perf tools and flamegraph.
ngx_shmtx_lock ngx_shmtx_unlock

strace信息：

perf top信息：

We have mounted too many routes, so the metrics are very large. The currently allocated memory is 100M.
what should i do ? scale prometheus share dict?

Environment

apisix :2.15
Linux 4 3.1-1160.92.1.el7.x86_64 #1 SMP Tue Jun 20 11:48:01 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
openresty/1.21.4.2

[zhaoqiang1980]

add etcd version ：3.5.0

[juzhiyuan]

Hi because your APISIX instance is too old (2.15), I can't provide any clues/solutions. If you can try to reproduce in the latest version, I can try to help :)

[zhaoqiang1980]

Hi because your APISIX instance is too old (2.15), I can't provide any clues/solutions. If you can try to reproduce in the latest version, I can try to help :)

In version 3.x, the dashboard has been integrated into the API. However, since we have enhanced our dashboard, upgrading to version 3 would require significant modifications. and we have more time and people to do this.

I also want to upgrade, but the gateway is used by the entire company, so we can't upgrade it casually. We can't push for the upgrade either, as there's no funding or time allocated to us.
please tell me what should i do？Thank you~~~

[bzp2010]

In principle, we do not provide support for older versions, especially for spans such as 2.x to 3.x. If the same issue cannot be reproduced on the master branch, it will basically not be addressed (or will be addressed with very low priority).
Generally, you may need to build an independent test environment using the master code repository or at least the latest release version, and conduct testing there. If the issue is reproduced, you can report that it should be fixed.

However, regarding the abnormal CPU consumption issue you mentioned caused by Prometheus, I have heard a little about it. This seems to be caused by some unexpected implementation or integration within the library, which is not easy to fix. The most economical and quick improvement is to first provide a larger shdict buffer.

You are using an extremely outdated version. Since the version you are using, we have introduced some improvements to the plugin, such as a mechanism for evicting old data (in the old implementation, shdict keys did not have a TTL, which allowed data to accumulate indefinitely).
Although abnormal CPU consumption may still occur if configured improperly (due to internal library details), the presence of shdict TTL can protect against most scenarios.

If you cannot upgrade, you will not be able to benefit from these improvements. If they are necessary, you will need to explore how to backport them to your version on your own, but this essentially remains a customization and will not receive official support, so experience is required.

[moonming]

@zhaoqiang1980 Thanks for the detailed report!

This is actually a known issue. It tends to occur more frequently when the shared dictionary (prometheus-metrics) is configured with a relatively small size. Under high concurrency and with a large number of metrics, the shared dict becomes a hotspot and introduces lock contention.

The most straightforward mitigation is to increase the size of the shared dict to reduce contention.

I think a more robust solution would be to implement a graceful degradation mechanism in the prometheus plugin. For example, when it detects that the shared memory is full and lock contention is impacting performance, it could temporarily pause metrics collection for 5 minutes. This may result in some metrics loss, but would prevent the CPU from hitting 100% and affecting overall system stability.

We’d love to hear what others think about this approach.

[zhaoqiang1980]

@zhaoqiang1980 Thanks for the detailed report!

This is actually a known issue. It tends to occur more frequently when the shared dictionary (prometheus-metrics) is configured with a relatively small size. Under high concurrency and with a large number of metrics, the shared dict becomes a hotspot and introduces lock contention.

The most straightforward mitigation is to increase the size of the shared dict to reduce contention.

I think a more robust solution would be to implement a graceful degradation mechanism in the prometheus plugin. For example, when it detects that the shared memory is full and lock contention is impacting performance, it could temporarily pause metrics collection for 5 minutes. This may result in some metrics loss, but would prevent the CPU from hitting 100% and affecting overall system stability.

We’d love to hear what others think about this approach.

---

@moonming Thanks for your response. we would increase the size of the shared dict. Otherwise we will attempt to introduce the higher version of the TTL capability into the current version to try to address the issue where metrics only increase and do not decrease. and thanks @bzp2010

[shlok-srivastava]

We hit the same failure mode and captured a CPU profile that confirms the mechanism, in case it helps others landing here.

Environment: APISIX 3.15.0 (official apache/apisix:3.15.0-ubuntu), openresty/1.27.1.2, traditional role, on a 6.12 kernel. Pods peg at their CPU limit after ~6–14h and start serving 504s.

Confirming your observation (CPU stays at 100% even with traffic removed)

We saw exactly this. We relabeled a pegged pod out of the Service so it received zero traffic, and it stayed pegged at the CPU ceiling for 15+ minutes — the peg is completely traffic-independent, driven by the ngx.timer counter-sync path.

perf record -F 99 --call-graph dwarf on one pegged worker (all workers were hot equally; master/privileged-agent idle):

75.6%  ngx_shmtx_lock              (self)
 6.2%  ngx_meta_lua_ffi_shdict_get
 2.8%  ngx_meta_lua_ffi_shdict_store
 4.0%  ngx_meta_lua_shdict_lookup
 4.3%  ngx_shmtx_unlock
 1.2%  ngx_meta_lua_shdict_expire

Hot stacks are all [JIT'd Lua: prometheus counter] → ngx_meta_lua_ffi_shdict_get/store → ngx_shmtx_lock. GC is negligible (libluajit ~0.9%). It happens once the prometheus-metrics shared dict is full (free_size → 0). That matches the flamegraph and strace in the original report exactly.

Root cause in our case (3.15+ specific)

The prometheus-metrics dict was filling because the built-in LLM histograms (llm_latency, llm_prompt_tokens, llm_completion_tokens) were being recorded on every request, including non-LLM traditional_http traffic

-- ngx_tpl.lua: set $llm_time_to_first_token '0';
-- exporter.lua (3.15/3.16):
if llm_time_to_first_token ~= "" then          -- '0' ~= "" is always true
    metrics.llm_latency:observe(...)            -- fires on every request
end

This is fixed in 3.17.0 (guard changed to ~= "0"). (Tracked in #12953 / #13088.)

Fixes that should work

• Disable prometheus plugin 🤷
• Upgrade to 3.17.0 (fixes the LLM-metric over-recording at the source).
• Or keep the plugin enabled and bound the cardinality: set plugin_attr.prometheus.metrics.<name>.expire (TTL to evict idle series), trim plugin_attr.prometheus.llm_latency_buckets, and enlarge the prometheus-metrics dict (default is only 10m).

Haven't tried the last one. I just chose to disable the plugin until we perform the upgrade.

[juzhiyuan]

Hi @shlok-srivastava, do you have plans to upgrade to 3.17 in Q3? Let us know if the issue gets fixed.

If no, I will request my team to try to reproduce and fix it further.

[shlok-srivastava]

We had upgraded to 3.17; around 20 hours later, the same issue happened again.
Have disabled Prometheus metrics now.

[nic-6443]

I spent some time on this and I believe the root cause is an infinite busy loop in the nginx-lua-prometheus key index. It exists unchanged from 2.15 (upstream 0.20220527) through 3.15 (nginx-lua-prometheus-api7 0.20240201), 3.17 (0.20250302) and current master (0.20260623) — neither #13088 nor #13602 touches it, which is consistent with @shlok-srivastava hitting it again after upgrading to 3.17.

The library tracks metric names in a key index stored in the same shared dict: slots __ngx_prom__key_1 .. key_N plus a counter __ngx_prom__key_count. Every new time series goes through KeyIndex:add() in prometheus_keys.lua:

while true do
  local N = self:sync()                    -- 2x dict:get (delete_count, key_count)
  if self.index[key] ~= nil then break end
  N = N + 1
  local ok, err = self.dict:add(self.key_prefix .. N, key)   -- 1x dict:add
  if ok then
    self.dict:incr(self.key_count, 1, 0)   -- key_count only advances on success
    ...
    break
  elseif err ~= "exists" then
    return "Unexpected error adding a key: " .. err
  end
  -- err == "exists": retry the same slot, forever
end

The loop only makes progress if slot key_count + 1 is free, and a full dict breaks that invariant. When a full dict needs room, lua-nginx-module force-evicts the LRU-tail node (ngx_http_lua_shdict_expire(ctx, 0), "deletes oldest entry by force") on every set/add/incr. __ngx_prom__key_count is an ordinary evictable node, and it is only refreshed when a worker registers/looks up a new key through the index — under steady traffic with a stable set of series nothing touches it, so it ages to the LRU tail and eventually gets evicted (that narrow timing window is also why this takes hours and only hits some instances — 2/38 above). After that sync() reads it as nil → 0, and if one add does succeed, dict:incr(key_count, 1, 0) re-creates it at 1, far below the surviving slots. The next new series then does dict:add("__ngx_prom__key_1", ...) against a slot that still exists → "exists" → same N → "exists" → forever. (The key_count rewind also silently corrupts the index — duplicate/missing series, the milder symptom family that 0.20260623 tried to address.)

Two properties make it exactly what you both captured:

• the loop never yields, so the worker never returns to the event loop: 100% CPU forever, completely traffic-independent — removing traffic can't help because the stuck request never finishes. Any other worker that later registers any new series joins the spin. Only a restart recovers: reload keeps the shm zone, and Prometheus.init() itself calls key_index:add() from init_worker, so reloaded workers can get stuck before serving a single request.
• every failed dict:add() does a lookup on the blocking slot, which moves that node to the LRU head — the node blocking progress becomes the most-recently-used one and can never be evicted. The poisoned state is self-sustaining.

The loop body is 2 gets + 1 store per iteration, and the flamegraph in the original report confirms it: extracting the numbers from the SVG gives ngx_meta_lua_ffi_shdict_get 65.78% vs ngx_meta_lua_ffi_shdict_store 32.68% — a 2.01:1 ratio, with ngx_shmtx_lock under both. There is no ffi_shdict_incr frame anywhere in that flamegraph, which rules out metric write pressure (counter/histogram flushes go through dict:incr): those workers weren't recording metrics at all, they were spinning in this loop. The remaining symbols are the loop too — ngx_crc32_short/ngx_memn2cmp (key hashing/comparison) and lj_strfmt_wint/lj_str_new (the key_prefix .. N concat each iteration).

Repro, takes seconds, no APISIX needed — drives the real library:

git clone https://github.com/api7/nginx-lua-prometheus
resty -I nginx-lua-prometheus --shdict 'prometheus_metrics 10m' repro.lua

repro.lua

local KeyIndex = require("prometheus_keys")
local dict = ngx.shared.prometheus_metrics

-- 50 series registered through the public API
local ki = KeyIndex.new(dict, "__ngx_prom__")
for i = 1, 50 do
  ki:add('apisix_http_status{route="' .. i .. '"}', "dict full")
end

-- Stand-in for the one thing the shared dict does on its own under memory
-- pressure: forcible eviction of the LRU-tail node, which is key_count
-- whenever no new series has been registered for a while.
dict:delete("__ngx_prom__key_count")

-- watchdog: abort instead of hanging the process forever
local count = 0
local proxy = setmetatable({}, { __index = function(_, method)
  return function(_, ...)
    if method == "add" then
      count = count + 1
      if count > 200000 then error("SPIN", 0) end
    end
    return dict[method](dict, ...)
  end
end })
ki.dict = proxy

-- one brand-new series arrives...
local t0 = ngx.now()
local ok, err = pcall(ki.add, ki, 'apisix_http_status{route="brand-new"}', "dict full")
ngx.update_time()
if not ok and err == "SPIN" then
  print(string.format("infinite loop: 200000 failed dict:add() on the same slot in %.2fs, add() never returns", ngx.now() - t0))
else
  print("add() returned normally")
end

Output on both 0.20250302 (3.17.0's dep) and current master 0.20260623: the watchdog fires after 200k failed add() in ~0.3s; without it the call never returns. The 2.15-era upstream lib has the identical loop, and so does upstream knyar/nginx-lua-prometheus master.

I also ran the uncapped version inside a 4-worker openresty (apisix-runtime) and profiled the stuck workers to compare with your captures:

perf (self time)      2.15 report   3.15 report   this repro
ngx_shmtx_lock           70.8%         75.6%         75.3%
ffi_shdict_get            3.9%          6.2%         11.2%
ffi_shdict_store          2.0%          2.8%          3.6%
shdict_lookup             2.3%          4.0%          2.0%
shdict_expire             2.2%          1.2%          1.1%
ngx_shmtx_unlock         10.1%          4.3%          5.6%

strace on a stuck worker shows the same futex(FUTEX_WAKE) / FUTEX_WAIT = EAGAIN flood as the original screenshot (the shdict mutex semaphore under contention), and the workers stayed pegged at ~100% with zero traffic until killed.

For the fix, I think KeyIndex:add() has to stop trusting key_count unconditionally: when add() returns "exists", key_count has fallen behind an occupied slot, so advance it / adopt the occupant into the local index instead of retrying the same slot forever — plus a hard iteration cap that degrades to the existing "dict full" error rather than spinning. I validated that locally: with the same poisoned state a guarded add() returns in ~3ms, places the key and repairs key_count.

Until then the practical mitigation is what @moonming said: keep prometheus-metrics large enough that it never actually fills (the spin can only arm on a full dict — worth alerting on free_space() reaching 0); expire/fewer labels/smaller bucket sets slow the fill but don't remove the loop.