"no process" error when a producer was trying to check out a process from the pool

[x-ji]

Hi, we encountered this weird error and we're not sure how to look into it further.

May 6, 2021 @ 17:05:03.765    lib/ex_rabbit_pool.ex:21: ExRabbitPool.get_connection_worker/1
May 6, 2021 @ 17:05:03.765    /opt/app/deps/poolboy/src/poolboy.erl:63: :poolboy.checkout/3
	May 6, 2021 @ 17:05:03.765
    (stdlib 3.13.2) gen_server.erl:246: :gen_server.call/3
May 6, 2021 @ 17:05:03.765    lib/ex_rabbit_pool.ex:36: ExRabbitPool.with_channel/2
	May 6, 2021 @ 17:05:03.765
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
May 6, 2021 @ 17:05:03.765** (exit) exited in: :gen_server.call(:assistant_producers_pool, {:checkout, #Reference<0.647931993.1703411713.0>, true}, 5000)
May 6, 2021 @ 17:05:03.765    (assistant 0.0.1) lib/assistant/attachments/events/received_parser_response.ex:15: Assistant.Attachments.Events.ReceivedParserResponse.emit_received_parser_response/1
May 6, 2021 @ 17:05:03.765    (assistant 0.0.1) lib/assistant/rabbit/producer/publisher.ex:12: Assistant.Rabbit.Producer.Publisher.publish/5

It seems (the interpretation might be wrong though) that the pool provided the producer with a reference, but when the producer actually tried to check out that reference, it cannot find the process, and thus it crashed.

Has anybody seen something similar before? We wonder what could be the potential cause that a connection process is no longer alive/cannot be found under the reference given by ex_rabbit_pool. Could it be that when the pool is being intensely used (it's a segment of code where a lot of events are being emitted), its resources could be starved/a race condition occurs?

The config to initialize the producer pool (which I'm not sure if contains anything out of the ordinary):

  def init_producer_pool(application) when is_atom(application) do
    rabbit_config = get_rabbit_config()

    producers_pool = [
      name: {:local, Helpers.build_producers_pool(application)},
      worker_module: ExRabbitPool.Worker.RabbitConnection,
      size: max(1, Producer.count_producers(application)),
      max_overflow: 0
    ]

    {:ok, _pid} =
      ExRabbitPool.PoolSupervisor.start_link(
        rabbitmq_config: [
          channels: 1,
          username: Keyword.fetch!(rabbit_config, :rabbit_user),
          password: Keyword.fetch!(rabbit_config, :rabbit_password),
          host: Keyword.fetch!(rabbit_config, :rabbit_host),
          port: Keyword.fetch!(rabbit_config, :rabbit_port)
        ],
        connection_pools: [producers_pool]
      )

    :ok
  end

Nevertheless, I wonder if tweaking some config (e.g. increase the size, the max_overflow (it's not clear from a simple search of the documentation what this is used for), or the channels could help?

Thanks.