OOMKilled with 30 repos/120 workflows

[gesabehrends]

First, thanks a lot for maintaining the gitactionboard it is a very handy project!

Describe the bug
I run in k8s with ~30 repos and in total ~120 workflows. The Memory limit is set to 3 Gi. I see memory increasing linearly over ~1h after pod start until the pod gets OOMKilled (while the dashboard is open and being refreshed).

Is this a known issue or expected scaling limitation with larger numbers of repositories/workflows? Do you know of any memory leaks or have any recommendations for settings?

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[sumanmaity112]

Hi @gesabehrends, we do make use of in-momery cache, but I don't expect to grow rapidly. But given you had faced OOM issue, I think we need dig deeper to understand and analyze it

[gesabehrends]

Weirdly enough I could not reproduce while running the service locally with the same repos and the same memory settings, but the caches looked sane in that set-up. I tried to give more memory to non-heap in prod which did not help and reduce the polling to every 2 min (which seems to be the local setting?) which also did not help. I think we had to disable virtual threads in prod.