I've managed to solve this but posting for anyone searching:
I installed LibreNMS docker on a Mac Mini M4 Pro running Docker Desktop. Leaving it polling 30 devices for a few hours, the time per device would creep from 10 seconds to 300 and then it'd just stop working. Web pages would still be served but no polling. Looking at netstat and lsof I could see lots of hung UDP sockets for each of the polled devices. It seems like the UDP sockets weren't being released.
I tried enabling kernel networking mode and this resolved it.