mix release boot scripts preload only the bootstrap module set, exposing release boots (eval/migrations) to an OTP kernel-start race

[bruckner]

Elixir and Erlang/OTP versions

Erlang/OTP 28 [erts-16.4] [64-bit] [smp] [jit]
Elixir 1.19.5 (compiled with Erlang/OTP 28)

The generated boot script shape is the same on current main. The
underlying OTP race is erlang/otp#11353 (fix proposed in
erlang/otp#11354) and affects at least OTP 27 and 28.

Operating system

Linux (Debian 13), x86_64 and arm64.

Current behavior

Boot scripts generated by mix release (start.script,
start_clean.script) contain a single primLoad with only the
~29-module bootstrap set (error_handler, code, code_server,
logger*, proc_lib, ...). Everything else — including stdlib
modules the logger needs while handling early kernel-start work
(maps, sys, queue, io_lib) — is lazy-loaded during boot. The
stock start_clean.script shipped inside an OTP install instead
preloads all of kernel+stdlib before any kernel process starts.

That difference exposes every mix-release boot to erlang/otp#11353:
code_server:init registers its name before publishing its mode in
persistent_term, and error_handler switches to
code:ensure_loaded/1 (which reads that term) as soon as the name
resolves — so a lazy load landing in that init window dies with a
persistent_term:get(code_server) badarg. When the loading process is
the logger kernel process, the boot dies:

Kernel pid terminated (logger) ({badarg,[{persistent_term,get,[code_server],...},
  {code_server,get_mode,0,...},{code,ensure_loaded,1,...},
  {error_handler,undefined_function,3,...},{logger_backend,call_handlers,3,...},
  {proc_lib,exit_p,3,...}]})

In production this hit us through bin/<release> eval — the standard
migration/maintenance path — failing fleet-wide on a host while the
running application stayed healthy. A deterministic trigger is any
early-boot logger event; e.g. running the release from a directory the
release user cannot read makes erl_prim_loader emit file_error
reports for the "." code-path entry on every lazy load (an easy state
to be in: cd /root && runuser -u app -- bin/app eval ...). With that
trigger the crash reproduces 5/5; with a readable cwd the same boot
passes 5/5. Full analysis, an Elixir-free reproduction script, and a
crash-dump anatomy are in erlang/otp#11353.

Expected behavior

Release boots should not be able to lazy-load, during the kernel-start
window, modules that OTP's own boot scripts always preload. Two
options, either of which closes the exposure regardless of when/whether
the OTP fix ships:

1. Generate primLoad phases for kernel+stdlib modules the way the
   stock start_clean.script does (load-before-kernel-start), or

2. Minimally, extend the bootstrap primLoad with the closure the
   early logger path can touch. The set we deployed and verified
   (crash 5/5 → pass, plus 3/3 on the originally affected host):

   maps sys queue binary gb_sets gb_trees beam_lib os io io_lib
   io_lib_format io_lib_pretty unicode string calendar proplists
   gen_statem logger_olp logger_proxy logger_h_common logger_std_h
   logger_formatter erl_features
   

Workaround we ship today, as a release step, in case it is useful to
anyone landing here (patches both generated scripts and regenerates the
.boot files):

# mix.exs — releases: [myapp: [steps: [:assemble, &preload_boot_modules/1, :tar]]]
def preload_boot_modules(%Mix.Release{version_path: vsn_path} = release) do
  for name <- ["start", "start_clean"],
      script = Path.join(vsn_path, name <> ".script"),
      File.exists?(script) do
    {:ok, [{:script, info, instrs}]} = :file.consult(script)

    {:done, patched} =
      Enum.reduce(instrs, {:pending, []}, fn
        {:primLoad, mods}, {:pending, acc} ->
          {:done, [{:primLoad, Enum.sort(Enum.uniq(mods ++ @boot_preload_modules))} | acc]}

        instr, {state, acc} ->
          {state, [instr | acc]}
      end)

    File.write!(script, :io_lib.format(~c"%% coding: utf-8~n~tp.~n", [
      {:script, info, Enum.reverse(patched)}
    ]))

    :ok = :systools.script2boot(script |> Path.rootname() |> String.to_charlist())
  end

  release
end

[josevalim]

@bruckner I believe the Erlang/OTP fix is good but I cannot reproduce this part of the issue:

Boot scripts generated by mix release (start.script, start_clean.script) contain a single primLoad with only the ~29-module bootstrap set (error_handler, code, code_server, logger*, proc_lib, ...). Everything else — including stdlib modules the logger needs while handling early kernel-start work (maps, sys, queue, io_lib) — is lazy-loaded during boot.

I have generated a bootscript and the beginning is exactly the same as Erlang/OTP's start_clean.script:

%% coding: utf-8
{script,
    {"eval","0.1.0"},
    [{preLoaded,
         [atomics,counters,erl_init,erl_prim_loader,erl_tracer,erlang,
          erts_code_purger,erts_dirty_process_signal_handler,erts_internal,
          erts_literal_area_collector,erts_trace_cleaner,init,persistent_term,
          prim_buffer,prim_eval,prim_file,prim_inet,prim_net,prim_socket,
          prim_zip,socket_registry,zlib]},
     {progress,preloaded},
     {path,["$ROOT/lib/kernel-10.4.1/ebin","$ROOT/lib/stdlib-7.1/ebin"]},
     {primLoad,
         [error_handler,application,application_controller,application_master,
          code,code_server,erl_eval,erl_lint,erl_parse,error_logger,ets,file,
          filename,file_server,file_io_server,gen,gen_event,gen_server,heart,
          kernel,logger,logger_filters,logger_server,logger_backend,
          logger_config,logger_simple_h,lists,proc_lib,supervisor]},
     {kernel_load_completed},
     {progress,kernel_load_completed},
     {path,["$ROOT/lib/kernel-10.4.1/ebin"]},
     {primLoad,
         [application_starter,auth,disk_log,disk_log_1,disk_log_server,
          disk_log_sup,dist_ac,dist_util,erl_boot_server,erl_compile_server,
          erl_ddll,erl_debugger,erl_distribution,erl_epmd,erl_erts_errors,
          erl_kernel_errors,erl_reply,erl_signal_handler,erpc,erts_debug,
          gen_sctp,gen_tcp,gen_tcp_socket,gen_udp,gen_udp_socket,global,
          global_group,global_search,group,group_history,inet,inet6_sctp,
          inet6_tcp,inet6_tcp_dist,inet6_udp,inet_config,inet_db,inet_dns,
          inet_dns_tsig,inet_epmd_dist,inet_epmd_socket,inet_gethost_native,
          inet_hosts,inet_parse,inet_res,inet_sctp,inet_tcp,inet_tcp_dist,
          inet_udp,kernel_config,kernel_refc,local_tcp,local_udp,
          logger_disk_log_h,logger_formatter,logger_h_common,logger_handler,
          logger_handler_watcher,logger_olp,logger_proxy,logger_std_h,
          logger_sup,net,net_adm,net_kernel,os,pg,pg2,prim_tty,
          prim_tty_sighandler,ram_file,raw_file_io,raw_file_io_compressed,
          raw_file_io_deflate,raw_file_io_delayed,raw_file_io_inflate,
          raw_file_io_list,rpc,seq_trace,socket,standard_error,trace,user_drv,
          user_sup,wrap_log_reader]},
     {path,["$ROOT/lib/stdlib-7.1/ebin"]},
     {primLoad,
         [argparse,array,base64,beam_lib,binary,c,calendar,dets,dets_server,
          dets_sup,dets_utils,dets_v9,dict,digraph,digraph_utils,edlin,
          edlin_context,edlin_expand,edlin_key,edlin_type_suggestion,epp,
          erl_abstract_code,erl_anno,erl_bits,erl_compile,erl_error,
          erl_expand_records,erl_features,erl_internal,erl_posix_msg,erl_pp,
          erl_scan,erl_stdlib_errors,erl_tar,error_logger_file_h,
          error_logger_tty_h,escript,eval_bits,file_sorter,filelib,gb_sets,
          gb_trees,gen_fsm,gen_statem,io,io_lib,io_lib_format,io_lib_fread,
          io_lib_pretty,json,log_mf_h,maps,math,ms_transform,orddict,ordsets,
          otp_internal,peer,pool,proplists,qlc,qlc_pt,queue,rand,random,re,
          sets,shell,shell_default,shell_docs,shell_docs_markdown,
          shell_docs_test,slave,sofs,string,supervisor_bridge,sys,timer,
          unicode,unicode_util,uri_string,win32reg,zip,zstd]},

Then it proceeds to primLoad all Elixir applications only then to call:

     {kernelProcess,heart,{heart,start,[]}},
     {kernelProcess,logger,{logger_server,start_link,[]}},
     {kernelProcess,application_controller,
         {application_controller,start,
             [{application,kernel,

Can you please describe how are you generating your release and how I can reproduce the limited versions that you mention?

[josevalim]

Btw, I did some digging and I don't think this is caused by Elixir, I have written the report here: erlang/otp#11353 (comment)

If you can show Elixir can indeed produce faulty script files, I will gladly reopen. Thank you! ❤️

[bruckner]

Confirmed on my side, and apologies for the noise: re-inspecting the
exact artifact that failed, its generated start_clean.script has all
67 primLoad blocks — full kernel and stdlib included, mirroring
yours. My report's claim was an inspection error (first-match-only
grep, plus -init_debug output that doesn't echo skipped
instructions), and the effective lazy-loading traces to OTP
interactive mode ignoring post-{kernel_load_completed} primLoads, as
you wrote on the OTP issue — stock erl reproduces without Elixir
anywhere (one-liner in erlang/otp#11353). Elixir's codegen is fine and
closing was right.

For anyone landing here before the OTP fix ships: the release-step
workaround in the issue body still works, because it extends the
bootstrap primLoad — the one block interactive mode does honor.