Reduce memory usage in IEx autocomplete module listing#15140
Conversation
Use :erlang.loaded/0 instead of :code.all_loaded/0 since we only need module names, not file paths. The latter copies the full path for every loaded module, which on production systems with many dependencies caused significant memory spikes (measured 2.8MB -> 172KB on a real pod). Also replace Enum.sort/1 |> Enum.dedup/1 with :lists.usort/1 to eliminate one intermediate list allocation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace String.split/Enum.at/length with binary_part and :binary.match to extract submodule names. This avoids allocating intermediate lists of string fragments for every matching module. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
NIce! iex> (:erlang.loaded() |> :lists.usort()) -- (:code.all_loaded() |> Enum.map(&elem(&1, 0)) |> :lists.usort())
[:elixir_compiler_0, :elixir_compiler_1, :elixir_compiler_2, :elixir_compiler_3] |
|
I only checked it like that: But still on the app I'm testing I'm getting an empty list. This is some older elixir version and a release running on kubernetes where this bothers me the most :D |
|
I think the compiler modules will be fine because are likely not looking up under |
|
💚 💙 💜 💛 ❤️ |
| parts = String.split(mod, "."), | ||
| depth <= length(parts), | ||
| name = Enum.at(parts, depth - 1), | ||
| rest = binary_part(mod, prefix_size, byte_size(mod) - prefix_size), |
There was a problem hiding this comment.
Sorry, there may be a bug here. is it guaranteeed that byte_size(mod) - prefix_size is greater than 0?
Yes I totally agree, my point was more about the fact these are not returning exactly the same thing, so I wondered if there was a risk of having a wonky behavior. But worst case scenario, we can always check for |
|
Oh, I see and good call! We can use module_loaded in the future if we run into something, yeah! |
Summary
Reduce memory usage in IEx autocomplete module listing. On memory-capped pods, tab completion could trigger OOM kills (exit 137) due to unnecessary allocations.
Two changes:
Use
:erlang.loaded/0instead of:code.all_loaded/0and:lists.usort/1instead ofEnum.sort/1 |> Enum.dedup/1. We only need module names for autocomplete, not file paths.:code.all_loaded/0copies the full beam file path for every loaded module, which is pure waste.:lists.usort/1combines sort + dedup in a single native pass.Use
binary_part+:binary.matchinstead ofString.split+Enum.atinmatch_elixir_modules. Avoids allocating intermediate lists of string fragments for every matching module.Measurements on a production pod (~200 Elixir modules)
14.7x reduction — from 15.6 MB to 1.1 MB per autocomplete call.
Test plan
make test_iex— 279 tests, 0 failures🤖 Generated with Claude Code