Increased memory use over time liked to log size growth

[jonwaland]

What installation are you running?

Production (netalertx) 📦

Is there an existing issue for this?

• I have searched the existing open and closed issues and I checked the docs https://docs.netalertx.com/

The issue occurs in the following browsers. Select at least 2.

• Firefox
• Chrome
• Edge
• Safari (unsupported) - PRs welcome
• N/A - This is an issue with the backend

Current Behavior

docker container memory usage is increasing.

(Built on: 04/05/2026, 15:03:34 | Version: v26.5.4 )

Expected Behavior

no memory increase.

Steps To Reproduce

No response

Relevant app.conf settings

#-----------------AUTOGENERATED FILE-----------------#
#                                                    #
#         Generated:  2026-05-04_18-16-25                 #
#                                                    #
#   Config file for the LAN intruder detection app:  #
#      https://github.com/netalertx/NetAlertX        #
#                                                    #
#-----------------AUTOGENERATED FILE-----------------#


# General
#---------------------------
LOADED_PLUGINS=['ARPSCAN','CSVBCKP','DBCLNP','INTRNT','MAINT','NEWDEV','NSLOOKUP','NTFPRCS','AVAHISCAN','SETPWD','SYNC','VNDRPDT','WORKFLOWS','UI','CUSTPROP','SMTP','ICMP']
LOADED_PLUGINS__metadata="{}"
DEEP_SLEEP=False
DEEP_SLEEP__metadata="{}"
DISCOVER_PLUGINS=True
DISCOVER_PLUGINS__metadata="{}"
SCAN_SUBNETS=['192.168.1.1-192.168.1.254 --interface=eno1']
SCAN_SUBNETS__metadata="{}"
LOG_LEVEL='minimal'
LOG_LEVEL__metadata="{}"
TIMEZONE='Australia/Sydney'
TIMEZONE__metadata="{}"
PLUGINS_KEEP_HIST=250
PLUGINS_KEEP_HIST__metadata="{}"
REPORT_DASHBOARD_URL='http://192.168.1.128:20211'
REPORT_DASHBOARD_URL__metadata="{}"
BACKEND_API_URL=''
BACKEND_API_URL__metadata="{}"
DAYS_TO_KEEP_EVENTS=90
DAYS_TO_KEEP_EVENTS__metadata="{}"
HRS_TO_KEEP_NEWDEV=0
HRS_TO_KEEP_NEWDEV__metadata="{}"
HRS_TO_KEEP_OFFDEV=0
HRS_TO_KEEP_OFFDEV__metadata="{}"
CLEAR_NEW_FLAG=0
CLEAR_NEW_FLAG__metadata="{}"
PRAGMA_JOURNAL_SIZE_LIMIT=50
PRAGMA_JOURNAL_SIZE_LIMIT__metadata="{}"
REFRESH_FQDN=False
REFRESH_FQDN__metadata="{}"
API_CUSTOM_SQL='SELECT * FROM Devices WHERE devPresentLastScan = 0'
API_CUSTOM_SQL__metadata="{}"
VERSION='v26.5.4'
VERSION__metadata="{}"
NETWORK_DEVICE_TYPES=['AP','Gateway','Firewall','Hypervisor','Powerline','Switch','WLAN','PLC','Router','USB LAN Adapter','USB WIFI Adapter','Internet']
NETWORK_DEVICE_TYPES__metadata="{}"
GRAPHQL_PORT=20212
GRAPHQL_PORT__metadata="{}"
API_TOKEN='t_X30IXgMdEDgfvbCDxeDA'
API_TOKEN__metadata="{}"

docker-compose.yml

services:
  netalertx:
    container_name: netalertx
    image: "ghcr.io/jokob-sk/netalertx:latest"     
    network_mode: "host"
    read_only: true
    cap_drop:
      - ALL
    cap_add:
      - NET_RAW
      - NET_ADMIN
      - NET_BIND_SERVICE
      - CHOWN
      - SETUID
      - SETGID
    restart: unless-stopped
    volumes:
      - /home/walaj/docker/netalertx:/data
      - type: bind                           # Bind mount for timezone consistency
        source: /etc/localtime
        target: /etc/localtime
        read_only: true
    environment:
      - PGID=1000
      - PUID=1000    
      - PORT=20211
    tmpfs:
#      - "/tmp:uid=20211,gid=20211,mode=1700,rw,noexec,nosuid,nodev,async,noatime,nodiratime"
      - "/tmp:mode=1700,uid=0,gid=0,rw,noexec,nosuid,nodev,async,noatime,nodiratime"
    dns:
      - 192.168.1.128
    # Resource limits to prevent resource exhaustion
    mem_limit: 2048m            # Maximum memory usage
    mem_reservation: 1024m      # Soft memory limit
    cpu_shares: 512             # Relative CPU weight for CPU contention scenarios
    pids_limit: 512             # Limit the number of processes/threads to prevent fork bombs
    logging:
      options:
        max-size: "10m"         # Rotate log files after they reach 10MB
        max-file: "3"           # Keep a maximum of 3 log files

Debug or Trace enabled

• I have read and followed the steps in the wiki link above and provided the required debug logs and the log section covers the time when the issue occurs.

Relevant app.log section

let me know if the log is likely to be useful

Docker Logs

 _   _      _    ___  _           _  __   __
| \ | |    | |  / _ \| |         | | \ \ / /
|  \| | ___| |_/ /_\ \ | ___ _ __| |_ \ V /
| .   |/ _ \ __|  _  | |/ _ \  __| __|/   \
| |\  |  __/ |_| | | | |  __/ |  | |_/ /^\ \
\_| \_/\___|\__\_| |_/_|\___|_|   \__\/   \/
   Network intruder and presence detector.
   https://netalertx.com


Startup pre-checks
--> data migration.sh
--> capabilities audit.sh
--> mounts.py
 Path                     | R | W | Mount | RAMDisk | Performance | DataLoss
--------------------------+---+---+-------+---------+-------------+----------
 /data                    | ✅| ✅|   ✅  |    ➖   |      ➖     |    ✅
 /data/db                 | ✅| ✅|   ✅  |    ➖   |      ➖     |    ✅
 /data/config             | ✅| ✅|   ✅  |    ➖   |      ➖     |    ✅
 /tmp/run/tmp             | ✅| ✅|   ✅  |    ✅   |      ✅     |    ✅
 /tmp/api                 | ✅| ✅|   ✅  |    ✅   |      ✅     |    ✅
 /tmp/log                 | ✅| ✅|   ✅  |    ✅   |      ✅     |    ✅
 /tmp/run                 | ✅| ✅|   ✅  |    ✅   |      ✅     |    ✅
 /tmp/nginx/active-config | ✅| ✅|   ✅  |    ✅   |      ✅     |    ✅
--> first run config.sh
--> first run db.sh
--> mandatory folders.sh
    * Creating NetAlertX log directory.
    * Creating NetAlertX API cache.
    * Creating System services runtime directory.
    * Creating nginx active configuration directory.
    * Creating Plugins log.
    * Creating System services run log.
    * Creating System services run tmp.
    * Creating DB locked log.
    * Creating Execution queue log.
--> apply conf override.sh
--> override individual settings.sh
--> host optimization.sh
--> writable config.sh
--> nginx config.sh
--> expected user id match.sh
--> host mode network.sh
--> excessive capabilities.sh
--> appliance integrity.sh
--> ports available.sh
Starting supercronic --quiet "/services/config/cron/crontab" >>"/tmp/log/cron.log" 2>&1 &
Starting /usr/sbin/php-fpm83 -y "/services/config/php/php-fpm.conf" -F (tee stderr to app.php_errors.log)
Starting python3  -m server > /tmp/log/stdout.log 2> >(tee /tmp/log/stderr.log >&2)
Starting /usr/sbin/nginx -p "/tmp/run/" -c "/tmp/nginx/active-config/nginx.conf" -g "error_log stderr; error_log /tmp/log/nginx-error.log; daemon off;" &
Successfully updated IEEE OUI database (114315 entries)

[jokob-sk]

Can you please include:

1. Screenshot of your Maintenance, DB tables sizes section
2. Verify the MAINT and DBCLNP plugins are running regularly (ideally post logs and corresponding RAM usage stats)

Increase in memory is expected with db growth to a given extend. See also docs for details : https://docs.netalertx.com/PERFORMANCE/

[jonwaland]

maint db tables screen grab:

searching app.log:
MAINT

12:01:15 [Plugins] No output received from the plugin "MAINT"
12:00:32 [Plugins] No output received from the plugin "MAINT"

DBCLNP

13:00:18 [Plugins] No output received from the plugin "DBCLNP"
13:31:16 [Plugins] No output received from the plugin "DBCLNP"
14:01:11 [Plugins] No output received from the plugin "DBCLNP"
14:31:07 [Plugins] No output received from the plugin "DBCLNP"
15:00:56 [Plugins] No output received from the plugin "DBCLNP"
15:30:51 [Plugins] No output received from the plugin "DBCLNP"
16:00:42 [Plugins] No output received from the plugin "DBCLNP"
16:30:36 [Plugins] No output received from the plugin "DBCLNP"

I've increased logging to debug....

[jonwaland]

16:58:18 [ 2026-06-02 06:58:18 ] START Run:  DBCLNP
16:58:18 [DBCLNP] In script
16:58:18 [DBCLNP] Upkeep Database: /data/db/app.db
16:58:20 [DBCLNP] REINDEX completed
16:58:20 [DBCLNP] Online_History: Delete all but keep latest 150 entries
16:58:20 [DBCLNP] Online_History deleted rows: 5
16:58:20 [DBCLNP] Events: Delete all older than 90 days (DAYS_TO_KEEP_EVENTS setting)
16:58:20 [DBCLNP] SQL : DELETE FROM Events WHERE eveDateTime <= date('now', '-90 day')
16:58:20 [DBCLNP] Events deleted rows: 0
16:58:20 [DBCLNP] Sessions: Delete all older than 90 days (reuses DAYS_TO_KEEP_EVENTS)
16:58:20 [DBCLNP] SQL : DELETE FROM Sessions WHERE sesDateTimeConnection <= date('now', '-90 day')
16:58:20 [DBCLNP] Sessions deleted rows: 0
16:58:20 [DBCLNP] Plugins_History: Trim to 250 per Plugin
16:58:20 [DBCLNP] Plugins_History deleted rows: 15
16:58:20 [DBCLNP] Notifications: Trim to 100
16:58:20 [DBCLNP] Notifications deleted rows: 0
16:58:20 [DBCLNP] Trim AppEvents to less than 5000
16:58:20 [DBCLNP] AppEvents deleted rows: 492
16:58:20 [DBCLNP] Plugins_Objects: Delete all duplicates
16:58:20 [DBCLNP] Plugins_Objects deleted rows: 0
16:58:20 [DBCLNP] WAL checkpoint executed to truncate file.
16:58:20 [DBCLNP] ANALYZE completed
16:58:20 [DBCLNP] Shrink Database
16:58:21 [Plugins] No output received from the plugin "DBCLNP"
16:58:21 [ 2026-06-02 06:58:21 ] END Run:  DBCLNP
16:58:21 [UserEventsQueueInstance] Processed event:  run
16:58:21 [check_and_run_user_event] INFO: Executed events:  run with param DBCLNP
16:59:22 [UserEventsQueueInstance] Action "run|MAINT" added to the execution queue.
127.0.0.1 - - [02/Jun/2026 16:59:22] "POST /logs/add-to-execution-queue HTTP/1.1" 200 -
16:59:26 [MAINT] In script
16:59:26 [MAINT] Cleaning file
16:59:26 [MAINT] app.log size BEFORE: 14104444
16:59:26 [MAINT] app.log size AFTER: 14104444
16:59:26 [MAINT] Cleanup of app.log finished
16:59:26 [MAINT] Cleaning file
16:59:26 [MAINT] nginx-error.log size BEFORE: 20217
16:59:26 [MAINT] nginx-error.log size AFTER: 20217
16:59:26 [MAINT] Cleanup of nginx-error.log finished
16:59:26 [MAINT] Cleaning file
16:59:26 [ 2026-06-02 06:59:26 ] START Run:  MAINT
16:59:26 [Plugins] No output received from the plugin "MAINT"
16:59:26 [ 2026-06-02 06:59:26 ] END Run:  MAINT
16:59:26 [UserEventsQueueInstance] Processed event:  run
16:59:26 [check_and_run_user_event] INFO: Executed events:  run with param MAINT

[jokob-sk]

Thanks, can you post corresponding RAM usage stats when the MAINT and DBCLNP plugins run? Did you try to adjust the WAL size setting as per docs?

[jonwaland]

RAM usage appears unaffected:

(CPU,MEM)
2026-06-02T16:40:31+10:00,netalertx,0.11%,22.18%
2026-06-02T16:41:33+10:00,netalertx,0.17%,22.31%
2026-06-02T16:42:35+10:00,netalertx,52.56%,22.33%
2026-06-02T16:43:37+10:00,netalertx,0.06%,22.31%
2026-06-02T16:44:39+10:00,netalertx,63.49%,22.31%
2026-06-02T16:45:40+10:00,netalertx,33.06%,22.32%
2026-06-02T16:46:42+10:00,netalertx,0.07%,22.32%
2026-06-02T16:47:44+10:00,netalertx,0.19%,22.32%
2026-06-02T16:48:45+10:00,netalertx,8.93%,22.50%
2026-06-02T16:49:47+10:00,netalertx,0.07%,22.49%
2026-06-02T16:50:49+10:00,netalertx,100.92%,22.50%
2026-06-02T16:51:50+10:00,netalertx,100.46%,22.54%
2026-06-02T16:52:52+10:00,netalertx,51.41%,22.57%
2026-06-02T16:53:54+10:00,netalertx,16.95%,22.52%
2026-06-02T16:54:56+10:00,netalertx,0.07%,22.32%
2026-06-02T16:55:57+10:00,netalertx,104.27%,22.62%
2026-06-02T16:56:59+10:00,netalertx,1.44%,22.45%
2026-06-02T16:58:01+10:00,netalertx,1.28%,22.34%   (CLEANUP/MAINT RUNNING)
2026-06-02T16:59:02+10:00,netalertx,0.06%,21.43%   (CLEANUP/MAINT RUNNING)
2026-06-02T17:00:04+10:00,netalertx,40.90%,21.45%
2026-06-02T17:01:06+10:00,netalertx,36.96%,23.23%
2026-06-02T17:02:07+10:00,netalertx,90.21%,22.43%
2026-06-02T17:03:09+10:00,netalertx,0.06%,22.20%
2026-06-02T17:04:11+10:00,netalertx,44.38%,22.19%
2026-06-02T17:05:13+10:00,netalertx,76.90%,22.25%

I've not touched the WAL sizing - its the default which with 88+ devices, seems to be correct. (50Mb WAL)

[jokob-sk]

The memory usage seems to be pretty stable at 22% unless I'm missing something from the above log?

[jokob-sk]

Also, what happened on May 16? Seems like the ram usage dropped around that time?

[jonwaland]

memory increase is at an average rate of 0.366% percentage points per day. This translates to a growth rate of approximately 2.56% percentage points per week.

May 16th I restarted the container, and same around 20th

[jokob-sk]

I've noticed your log files are not being trimmed:

16:59:26 [MAINT] app.log size BEFORE: 14104444
16:59:26 [MAINT] app.log size AFTER: 14104444
16:59:26 [MAINT] Cleanup of app.log finished

Can you decrease the log size setting MAINT_LOG_LENGTH to e.g. 1000 lines and see if there is a difference?

Also can you paste the output of ps -o pid,rss,vsz,time,comm when executing in the container?

I also don't see the following entry in your log:

14:24:52 [MAINT] stdout.log size BEFORE: 3106956
14:24:52 [MAINT] stdout.log size AFTER: 211197
14:24:52 [MAINT] Cleanup of stdout.log finished

Can you check the stdout.log file is cleaned after running the plugin?

[jonwaland]

MAINT_LOG_LENGTH changed from 10000000 to 1000 and MAINT run manually:

08:59:06 [UserEventsQueueInstance] Action "run|MAINT" added to the execution queue.
127.0.0.1 - - [03/Jun/2026 08:59:06] "POST /logs/add-to-execution-queue HTTP/1.1" 200 -
08:59:10 [Config] No old column references found in config. No changes made.
08:59:10 [Import Config] checking config file 
08:59:10 [Import Config] lastImportedConfFile     : 1780441132.5482037
08:59:10 [Import Config] fileModifiedTime         : 1780441132.5482037
08:59:10 [Import Config] skipping config file import
08:59:10 [check_and_run_user_event] Process User Execution Queue:[2026-06-02 22:59:06]|eda511d4-b612-49f9-84d0-0c701dc0f54a|run|MAINT

08:59:10 [ 2026-06-02 22:59:10 ] START Run:  MAINT
08:59:10 [Plugins] Timeout: 300
08:59:10 [Plugin utils] Pre-Resolved CMD:  python3 /app/front/plugins/maintenance/maintenance.py
08:59:10 [Plugins] Executing: python3 /app/front/plugins/maintenance/maintenance.py
08:59:10 [MAINT] In script
08:59:10 [MAINT] Cleaning file
08:59:10 [MAINT] app.log size BEFORE: 57727755
08:59:10 [MAINT] app.log size AFTER: 108326
08:59:10 [MAINT] Cleanup of app.log finished
08:59:10 [MAINT] Cleaning file
08:59:10 [MAINT] nginx-error.log size BEFORE: 25973
08:59:10 [MAINT] nginx-error.log size AFTER: 25973
08:59:10 [MAINT] Cleanup of nginx-error.log finished
08:59:10 [MAINT] Cleaning file
08:59:10 [Plugins] Resolved : ['python3', '/app/front/plugins/maintenance/maintenance.py']
08:59:11 [Plugins] Output: 08:59:10 [MAINT] In script
08:59:10 [MAINT] Cleaning file
08:59:10 [MAINT] app.log size BEFORE: 57727755
08:59:10 [MAINT] app.log size AFTER: 108326
08:59:10 [MAINT] Cleanup of app.log finished
08:59:10 [MAINT] Cleaning file
08:59:10 [MAINT] nginx-error.log size BEFORE: 25973
08:59:10 [MAINT] nginx-error.log size AFTER: 25973
08:59:10 [MAINT] Cleanup of nginx-error.log finished
08:59:10 [MAINT] Cleaning file
08:59:10 [MAINT] stdout.log size BEFORE: 55970355
08:59:11 [MAINT] stdout.log size AFTER: 108504
08:59:11 [MAINT] Cleanup of stdout.log finished

08:59:11 [Plugins] No output received from the plugin "MAINT"

output of ps -o pid,rss,vsz,time,comm from within container:

PID   RSS  VSZ  TIME  COMMAND
    1 1836 2444  0:02 root-entrypoint
   42 1848 2440  6:58 entrypoint.sh
  188 1816 2352  0:00 start-cron.sh
  189 1820 2356  0:00 start-php-fpm.s
  190 1796 2360  0:00 start-nginx.sh
  191 207m 223m  2d14 python3
  206  12m 1.1g  0:48 supercronic
  209  11m  23m  0:50 php-fpm83
  210    4 1628  0:00 tee
  215    0    0  0:00 cat
  216    4 1628  0:00 tee
  232 6732  12m  0:00 nginx
  234 3672  12m  0:26 nginx
  235 3680  12m  0:29 nginx
  236 3672  12m  0:26 nginx
  237 3656  12m  0:37 nginx
  238 3696  12m  0:29 nginx
  239 3648  12m  0:35 nginx
  240 3668  12m  0:26 nginx
  241 3704  12m  0:25 nginx
1708452  11m  24m  0:11 php-fpm83
1709307  11m  24m  0:00 php-fpm83
1803780  10m  24m  0:00 php-fpm83
1804070    4 1624  0:00 sleep
1804071    4 1632  0:00 ps

Which suggests the memory usage is linked to log length (the drop was at the time the logs were truncated - no drop was see when running MAINT without truncation)?

[jokob-sk]

Thanks @jonwaland for the tests. Looks like it. I'm thinking if I should decrease the default value of MAINT_LOG_LENGTH to e.g. 10000 to keep memory usage in an expected range. What do you think? I will also update the https://docs.netalertx.com/PERFORMANCE/ docs.

[jonwaland]

Certainly it seems that the size of the log file will impact the amount of memory used (not sure why though), and so if you have an increasing log file it will increase memory foot print. So yes - set default to something reasonable, and add a warning to the performance page