Python bindings: returned vertex/zone Arrays are never freed

[ajfriend]

In the Python bindings, the eC Array returned by getZoneRefinedWGS84Vertices is never freed, so long-running processes leak memory on every call. getZoneWGS84Vertices and listZones leak the same way. The cause seems to be that the wrapper classes have a delete() method but no __del__, so Python's garbage collector never releases the native side of any object the bindings create (GeoPoint, etc.).

import resource
import sys

import dggal as al
al.pydggal_setup(al.Application())


def mem_usage_mb():
    """Peak memory usage (max RSS) of this process, in MB."""
    scale = 1 if sys.platform == 'darwin' else 1024  # ru_maxrss: bytes on macOS, KB on Linux
    return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * scale / 2**20


isea = al.ISEA3H()
z = isea.getZoneFromWGS84Centroid(8, al.GeoPoint(42.1, 12.0))

print(f'     0 calls  mem = {mem_usage_mb():5.0f} MB')
for i in range(1, 50_001):
    isea.getZoneRefinedWGS84Vertices(z, 20)
    if i % 10_000 == 0:
        print(f'{i:6d} calls  mem = {mem_usage_mb():5.0f} MB')

Output:

     0 calls  mem =    28 MB
 10000 calls  mem =    60 MB
 20000 calls  mem =   100 MB
 30000 calls  mem =   137 MB
 40000 calls  mem =   139 MB
 50000 calls  mem =   193 MB

The workaround is explicitly freeing each returned array after copying the values out. However, arr.delete() doesn't seem to work, but ecrt.Instance.delete(arr) does. But maybe I'm doing something wrong?

The workaround comes with a tradeoff. The delete itself slows down dramatically as allocations accumulate (I can file another issue on that if it's helpful), so in bulk workloads it trades the leak for a slowdown.

[jerstlouis]

However, arr.delete() doesn't seem to work, but ecrt.Instance.delete(arr) does. But maybe I'm doing something wrong?

I believe the reason is that arr.delete() clashes with the delete method to delete an element of a Container:

So ecrt.Instance.delete(arr) is the correct thing to do to force a deletion of the array/container itself (e.g., as I do here).

The bindings don't define any __del__, but I'm not sure that is the cause of the leak, and whether that would be the proper thing to do for the bindings? We can explore what the best solution is.
One potential issue with the bindings is that the native eC side might still hold a reference, though potentially this could be solved by freeing the native side first from that...

The delete itself slows down dramatically as allocations accumulate (I can file another issue on that if it's helpful), so in bulk workloads it trades the leak for a slowdown.

I would be curious to know more about what exactly causes the slowdown.