Potential memory leak of 312 bytes when using the SSE feature

[stephanpelikan]

Platform

ESP8266

IDE / Tooling

Arduino (IDE/CLI)

What happened?

I'm using SSE feature. It seems, there is a leek of 312 bytes for each client connected. I've setup a minimum application (see MRE). It prints the HEAP every minute and also prints clients connecting and disconnecting:

12:06:08.501 -> Connected to WiFi: 192.168.88.27
12:07:04.537 -> 45992
12:08:04.544 -> 45992
12:08:41.651 -> Connect client: 42510 -> 1
12:09:04.577 -> 45464
12:10:04.571 -> 45464
12:10:05.887 -> Disconnect client: 0 -> 0
12:11:04.556 -> 45680
12:12:04.565 -> 45680
12:12:17.605 -> Connect client: 43500 -> 1
12:13:04.565 -> 45152
12:13:06.468 -> Disconnect client: 0 -> 0
12:14:04.582 -> 45368
12:15:04.576 -> 45368

As you can see the difference of heap before a client is connected and after a client disconnected is always 312 bytes never returned to heap. With this leak after 144 clients connected the system will crash. I guess even earlier because actions taken by clients also need heap temporarily. It is totally OK to spent a fraction of the heap for clients but this leak seems to be linear.

Is there something I'm doing wrong or is this a bug?

Versions used:

• ESP Async TCP 2.0.0
• ESP Async Webserver 3.7.7

Stack Trace

I'v a menu "Debug level" in ArduinoIDE but setting to any value like "CORE" or "HTTP_CLIENT+HTTP_SERVER" did not change the output in serial monitor which above.

Minimal Reproductible Example (MRE)

I created a tiny React web-application using the SSE: data.zip.
It is uploaded using the ArduinoIDE plugin as shown here.

This is the ESP8266 application the web-client connects to:

#define WIFI_SSID "YOUR-SSID"
#define WIFI_PASSWORD "YOUR-PWD"
#define WWW_USERNAME "username"
#define WWW_PASSWORD "password"

#include <ESP8266WiFi.h>
#include <ESPAsyncTCP.h>
#include <ESPAsyncWebServer.h>
#include "LittleFS.h"

AsyncWebServer httpRestServer(80);
WiFiEventHandler wifiConnectHandler;
WiFiEventHandler wifiDisconnectHandler;
AsyncEventSource statusEvents("/api/status-events");
uint8_t numberOfStatusEventsClients = 0;

void webappInitStatus(AsyncEventSourceClient *client) {

  if(client->lastId()){
    Serial.printf_P(PSTR("Client reconnected! Last message ID that it got is: %u\n"), client->lastId());
  }
  if (client->connected()) {
    numberOfStatusEventsClients += 1;
    Serial.printf_P(PSTR("Connect client: %u -> %u\n"), client->client()->remotePort(), numberOfStatusEventsClients);
    client->client()->onDisconnect(disconnectStatusClient, NULL);
    updateStatusClients();
  }

}

void updateStatusClients() {

  statusEvents.send(F("{}"), F("INIT"), millis());

}

void disconnectStatusClient(void *arg, AsyncClient *client) {

  numberOfStatusEventsClients -= 1;
  Serial.printf_P(PSTR("Disconnect client: %u -> %u\n"), client->remotePort(), numberOfStatusEventsClients);

}

void handleNotFound(AsyncWebServerRequest *request) {

  Serial.println("Not found");
  String message = F("File Not Found\n\n");
  message += F("URI: ");
  message += request->url();
  message += F("\nMethod: ");
  message += request->methodToString();
  message += "\n";
  request->send(404, F("text/plain"), message);

}

void onWifiConnect(const WiFiEventStationModeGotIP& event) {

  Serial.print(F("Connected to WiFi: "));
  Serial.println(WiFi.localIP().toString());

  httpRestServer.onNotFound(handleNotFound);
  httpRestServer
      .serveStatic("/", LittleFS, "/www/")
      .setDefaultFile("index.html")
      .setCacheControl("no-cache, no-store, max-age=0")
      .setAuthentication(WWW_USERNAME, WWW_PASSWORD);
  statusEvents.onConnect(webappInitStatus);
  httpRestServer.addHandler(&statusEvents);
  httpRestServer.begin();

}

void onWifiDisconnect(const WiFiEventStationModeDisconnected& event) {

  httpRestServer.end();

}

void setup() {

  Serial.begin(115200);
  Serial.println("\n\n\n\nStarted\n\n\n\n");

  if(!LittleFS.begin()) {
    Serial.println(F("An error has occurred on mounting LittleFS"));
    return;
  }

  wifiConnectHandler = WiFi.onStationModeGotIP(onWifiConnect);
  wifiDisconnectHandler = WiFi.onStationModeDisconnected(onWifiDisconnect);

  WiFi.persistent(false);
  WiFi.setAutoReconnect(false); // reconnect is done manually every minute
  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);

}

unsigned long previousTime = 0;

void loop() {

  unsigned long currentMillis = millis();
  if (currentMillis < previousTime) { // handle overflow
    previousTime = currentMillis;
    return;
  }
  if (currentMillis - previousTime < 60000) {
    return;
  }
  previousTime = currentMillis;

  Serial.println(ESP.getFreeHeap(), DEC);

  updateStatusClients(); // ping SSE clients

}

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of ESPAsyncWebServer (and AsyncTCP for ESP32).

[mathieucarbou]

Hello @stephanpelikan : you would need to determine first if the leak is caused by ESPAsyncTCP (ESP8266) or AsyncTCP (ESP32) or ESPAsyncWebServer.

We do not support ESP8266 anymore. No dev and no fix on ESPAsyncTCP. anymore

Also, FYI, on ESP32, there is also a decrease of heap as soon as LWIP layers are called.
And on connection close, it can take up to a minute to recover the heap used by a client.
A little less for aborted connections.

So for this issue:

1. you would need to reproduce it on ESP32
2. you would need to show that each SSE client connect / disconnect causes a heap memory leak which is not recovering after a while also on ESP32

Until then, there is nothing to work on on our side, sorry...

[stephanpelikan]

I ordered an ESP32 and will try to reproduce it.

An additional note that may be of interest to other readers:

I noticed already crashes when connecting a client from time to time. I couldn't figure out why this sometimes happens and why not. Yesterday, I kept the site open in Chrome and noticed a connection loss of SSE request every minute (approximately) which is recovered by the client. In this scenario each lost connection causes a heap loss. When accessing the site from a second device, shipping the site takes to much memory and it crashes.

As a workaround I send a "ping" message every 30 seconds. This keeps the connection open and heap loss only occurs when closing the site in the browser.

Heap loss without the workaround:

13:36:14.165 -> Waterlevel 100%
13:36:17.156 -> Connected to WiFi: 192.168.88.27
13:37:00.013 -> 43112
13:37:59.990 -> 43112
13:38:17.337 -> Not found (favicon.ico)
13:39:00.034 -> 42800
...
15:30:00.168 -> 9728
15:31:00.177 -> 9416
15:32:00.157 -> 9104
15:33:00.178 -> 8792
15:34:00.191 -> 8480
15:35:00.183 -> 8168
15:36:00.169 -> 7856
15:37:00.184 -> 8072
15:39:00.199 -> 7232
15:40:00.188 -> 6920
15:41:00.196 -> 6608
15:42:00.175 -> 6296
15:43:00.191 -> 5984
15:44:00.208 -> 5480
15:44:48.007 -> Not found (favicon.ico)
15:44:48.789 -> 
15:44:48.789 -> User exception (panic/abort/assert)
15:44:48.789 -> --------------- CUT HERE FOR EXCEPTION DECODER ---------------
15:44:48.789 -> 
15:44:48.789 -> Unhandled C++ exception: OOM

Heap loss having the workaround in place:

06:59:00.344 -> 43024
07:00:00.357 -> 43024
07:01:00.352 -> 43024
07:01:26.003 -> Not found (favicon.ico)
07:01:26.459 -> Connect client: 36002 -> 1
07:02:00.372 -> 42712
07:03:00.473 -> 42712
07:04:00.351 -> 42712
07:05:00.894 -> 42712
07:06:00.940 -> 42712
...
07:12:00.938 -> 42712
07:13:00.914 -> 42712
07:14:00.907 -> 42712
07:15:00.917 -> 42712
07:15:18.657 -> Disconnect client: 0 -> 0
07:16:00.916 -> 42928
07:17:00.911 -> 42928

Actually, there is memory freed. Not all, but more than without.

[mathieucarbou]

@stephanpelikan : what you can do is also try with previous versions of ESPAsyncWebServer.
Dev is done on ESP32. We keep compilation compatibility with esp8266 but we do not test on esp8266 neither fix.
So it is possible that some recent changes have an effect.

[stephanpelikan]

@mathieucarbou: I received the ESP32 32u and ported the software. The leak is still there. With ESP32 it is 432 bytes. First connection consumed more (1648 bytes) what is fine, but subsequent were 432 each and last 440 bytes:

13:46:56.309 -> Connected to WiFi: 192.168.88.52
13:47:43.791 -> 211044
13:48:43.798 -> 211044
13:49:43.785 -> 211076
13:50:43.795 -> 211076
13:50:53.341 -> Connect client: 49052 -> 1
13:51:43.781 -> 209168
13:52:43.789 -> 209168
13:52:51.400 -> Disconnect clien: 0 -> 0
13:53:43.774 -> 209428
13:54:43.765 -> 209428
13:55:43.778 -> 209428
13:56:14.797 -> Connect client: 4304 -> 1
13:56:43.765 -> 208736
13:57:21.727 -> Disconnect client: 0 -> 0
13:57:43.792 -> 208996
13:58:43.781 -> 208996
13:58:50.501 -> Connect client: 44572 -> 1
13:59:43.774 -> 208304
14:00:43.796 -> 208304
14:00:58.515 -> Disconnect client: 0 -> 0
14:01:43.798 -> 208564
14:02:43.794 -> 208564
14:03:43.816 -> 208564
14:04:43.784 -> 208564
14:05:43.794 -> 208564
14:06:01.305 -> Connect client: 3780 -> 1
14:06:43.778 -> 207864
14:07:43.776 -> 207864
14:08:07.273 -> Disconnect clent: 0 -> 0
14:08:43.798 -> 208124
14:09:43.788 -> 208124
14:10:43.815 -> 208124

I also did further tests: Sometimes the leak is 600, 564 bytes, sometimes, 400, 436 or 432 bytes per client connection.

Versions used:

1. Async TCP 3.4.0
2. ESP Async Webserver 3.7.7

This is the code used:

#define WIFI_SSID "YOUR-SSID"
#define WIFI_PASSWORD "YOUR-PWD"
#define WWW_USERNAME "username"
#define WWW_PASSWORD "password"

#include <esp_wifi.h>
#include <ESPAsyncWebServer.h>
#include "LittleFS.h"

AsyncWebServer httpRestServer(80);
WiFiEventId_t wifiConnectEvent;
WiFiEventId_t wifiDisconnectEvent;
AsyncEventSource statusEvents("/api/status-events");
uint8_t numberOfStatusEventsClients = 0;

void webappInitStatus(AsyncEventSourceClient *client) {

  if(client->lastId()){
    Serial.printf_P(PSTR("Client reconnected! Last message ID that it got is: %u\n"), client->lastId());
  }
  if (client->connected()) {
    numberOfStatusEventsClients += 1;
    Serial.printf_P(PSTR("Connect client: %u -> %u\n"), client->client()->remotePort(), numberOfStatusEventsClients);
    client->client()->onDisconnect(disconnectStatusClient, NULL);
    updateStatusClients();
  }

}

void updateStatusClients() {

  statusEvents.send(F("{}"), F("INIT"), millis());

}

void disconnectStatusClient(void *arg, AsyncClient *client) {

  numberOfStatusEventsClients -= 1;
  Serial.printf_P(PSTR("Disconnect client: %u -> %u\n"), client->remotePort(), numberOfStatusEventsClients);

}

void handleNotFound(AsyncWebServerRequest *request) {

  Serial.println("Not found");
  String message = F("File Not Found\n\n");
  message += F("URI: ");
  message += request->url();
  message += F("\nMethod: ");
  message += request->methodToString();
  message += "\n";
  request->send(404, F("text/plain"), message);

}

void onWifiConnect(WiFiEvent_t event, WiFiEventInfo_t info) {

  Serial.print(F("Connected to WiFi: "));
  Serial.println(WiFi.localIP().toString());

  httpRestServer.onNotFound(handleNotFound);
  httpRestServer
      .serveStatic("/", LittleFS, "/www/")
      .setDefaultFile("index.html")
      .setCacheControl("no-cache, no-store, max-age=0")
      .setAuthentication(WWW_USERNAME, WWW_PASSWORD);
  statusEvents.onConnect(webappInitStatus);
  httpRestServer.addHandler(&statusEvents);
  httpRestServer.begin();

}

void onWifiDisconnect(WiFiEvent_t event, WiFiEventInfo_t info) {

  Serial.println(F("Disconnected from WiFi, will reconnect"));
  httpRestServer.end();

  delay(1000);
  reconnect();

}

void reconnect() {

  WiFi.persistent(false);
  WiFi.setAutoReconnect(false); // reconnect is done manually every minute
  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  WiFi.config(INADDR_NONE, INADDR_NONE, INADDR_NONE);

}

void setup() {

  Serial.begin(115200);
  Serial.println("\n\n\n\nStarted\n\n\n\n");

  if(!LittleFS.begin()) {
    Serial.println(F("An error has occurred on mounting LittleFS"));
    return;
  }

  WiFi.setScanMethod(WIFI_ALL_CHANNEL_SCAN);
  WiFi.setSortMethod(WIFI_CONNECT_AP_BY_SIGNAL);
  wifiConnectEvent = WiFi.onEvent(onWifiConnect, WiFiEvent_t::ARDUINO_EVENT_WIFI_STA_GOT_IP);
  wifiDisconnectEvent = WiFi.onEvent(onWifiDisconnect, WiFiEvent_t::ARDUINO_EVENT_WIFI_STA_DISCONNECTED);

  WiFi.disconnect(true, true);
  delay(1000);
  reconnect();

}

unsigned long previousTime = 0;
unsigned long previousPing = 0;

void loop() {

  unsigned long currentMillis = millis();
  if (currentMillis < previousTime) { // handle overflow
    previousTime = currentMillis;
    previousPing = currentMillis;
    return;
  }
  if (currentMillis - previousPing > 30000) {
    statusEvents.send(F("{}"), F("PING"), currentMillis);
    previousPing = currentMillis;
  }
  if (currentMillis - previousTime < 60000) {
    return;
  }
  previousTime = currentMillis;

  Serial.println(ESP.getFreeHeap(), DEC);

}

[mathieucarbou]

I will have a deeper look when I will have more time.

fyi, only httpRestServer.begin(); has to be in onWiFiConnect. Remaining is setup to run only once.

[mathieucarbou]

@stephanpelikan : I think I know what is happening...

When you do that: client->client()->onDisconnect(...) you are overriding the callback that ESPAsyncWS sets on the AsyncTCP client line 180 to be able to correctly cleanup the client and data:

  _client->onDisconnect(
    [this](void *r, AsyncClient *c) {
      static_cast<AsyncEventSourceClient *>(r)->_onDisconnect();
      delete c;
    },
    this
  );

So these cleanup lines are not called:

static_cast<AsyncEventSourceClient *>(r)->_onDisconnect();
delete c;

Which obviously leaks memory.

Please have a look at the ServerSentEvents example in the project.
You must NEVER use the AsyncTCP client() internals.
Please use instead the callbacks provided by the AsyncEventSource object:

  events.onConnect([](AsyncEventSourceClient *client) {
    Serial.printf("SSE Client connected! ID: %" PRIu32 "\n", client->lastId());
    client->send("hello!", NULL, millis(), 1000);
  });

  events.onDisconnect([](AsyncEventSourceClient *client) {
    Serial.printf("SSE Client disconnected! ID: %" PRIu32 "\n", client->lastId());
  });