When running the reproduce script below (which causes intopt to raise ValueError), traceback, frame (codename: ecyglpki.Problem.intopt (ecyglpki.c:23524)), frame (codename: _leaky) objects are leaked. (The memory leaked is actually far greater than what the output reports below, it probably does not include C allocated memory).
Examining the referrers tree, frame(_leaky) is reffered to by frame(intopt) and traceback(1). frame(intopt) is reffered to by traceback(2); and traceback(2) is referred to by traceback(1) forming a cycle. Though, Python's GC supposedly handles cycles. If you run the reproduce script with iterations set to 5, you should get a Tk GUI allowing you to browse the referrer tree to the very_leaky frame.
Output (from reproduce script below). First object listing is before running intopt (multiple times). Second is directly after it. Last listing is after running gc.collect:
types | # objects | total size
==================================== | =========== | ============
<class 'dict | 2937 | 2.30 MB
<class 'type | 864 | 858.94 KB
<class 'set | 542 | 185.56 KB
<class 'tuple | 2432 | 165.21 KB
<class 'weakref | 1632 | 127.50 KB
<class 'wrapper_descriptor | 1548 | 120.94 KB
<class 'list | 739 | 111.91 KB
<class 'method_descriptor | 1286 | 90.42 KB
<class 'builtin_function_or_method | 1173 | 82.48 KB
<class 'getset_descriptor | 1131 | 79.52 KB
<class 'collections.OrderedDict | 69 | 75.62 KB
<class 'abc.ABCMeta | 74 | 72.48 KB
function (__init__) | 435 | 57.77 KB
<class 'frozenset | 62 | 37.81 KB
<class 'member_descriptor | 432 | 30.38 KB
types | # objects | total size
============================================================== | =========== | ============
frame (codename: _leaky) | 50000 | 24.80 MB
frame (codename: ecyglpki.Problem.intopt (ecyglpki.c:23524)) | 50000 | 19.07 MB
<class 'traceback | 100000 | 6.10 MB
<class 'dict | 2856 | 2.21 MB
<class 'type | 864 | 858.94 KB
<class 'list | 5460 | 554.85 KB
<class 'set | 542 | 185.56 KB
<class 'tuple | 2348 | 155.85 KB
<class 'weakref | 1632 | 127.50 KB
<class 'wrapper_descriptor | 1548 | 120.94 KB
<class 'method_descriptor | 1286 | 90.42 KB
<class 'builtin_function_or_method | 1173 | 82.48 KB
<class 'getset_descriptor | 1131 | 79.52 KB
<class 'collections.OrderedDict | 69 | 75.62 KB
<class 'abc.ABCMeta | 74 | 72.48 KB
gc.collect
types | # objects | total size
============================================================== | =========== | ============
frame (codename: _leaky) | 50000 | 24.80 MB
frame (codename: ecyglpki.Problem.intopt (ecyglpki.c:23524)) | 50000 | 19.07 MB
<class 'traceback | 100000 | 6.10 MB
<class 'dict | 2856 | 2.21 MB
<class 'type | 864 | 858.94 KB
<class 'list | 5464 | 555.20 KB
<class 'set | 542 | 185.56 KB
<class 'tuple | 2348 | 155.85 KB
<class 'weakref | 1632 | 127.50 KB
<class 'wrapper_descriptor | 1548 | 120.94 KB
<class 'method_descriptor | 1286 | 90.42 KB
<class 'builtin_function_or_method | 1173 | 82.48 KB
<class 'getset_descriptor | 1131 | 79.52 KB
<class 'collections.OrderedDict | 69 | 75.62 KB
<class 'abc.ABCMeta | 74 | 72.48 KB
To reproduce: python script.py the script below:
import numpy as np
import ecyglpki
nan = np.nan
nutrition_target = np.array([[ 0.57142857, 1.42857143],
[ 0.16513761, 1.83486239],
[ 0.57142857, 1.42857143],
[ 0.30188679, 1.69811321],
[ 0.90909091, nan],
[ 0.34586466, 1.65413534],
[ 0.29787234, 1.70212766],
[ 0.24175824, 1.75824176],
[ 0.43137255, 1.56862745],
[ 0.90909091, nan],
[ 0.78947368, 1.21052632],
[ 0.90909091, nan],
[ nan, 2. ],
[ 0.0861244 , 1.9138756 ],
[ 0.26086957, 1.73913043],
[ 0.90909091, nan],
[ nan, 2. ],
[ 0.90909091, nan],
[ 0.90909091, nan],
[ 0.90909091, nan],
[ 0.90909091, nan],
[ 0.02566634, 1.97433366],
[ 0.57142857, 1.42857143],
[ nan, 2. ],
[ 0.90909091, nan],
[ 0.90909091, nan],
[ 0.27160494, 1.72839506],
[ 0.90909091, nan],
[ 0.90909091, nan],
[ 0.90909091, nan],
[ 0.90909091, nan],
[ 0.90909091, nan],
[ 0.98591549, 1.01408451],
[ nan, 2. ],
[ nan, 2. ],
[ nan, 2. ],
[ nan, 2. ],
[ 0.81818182, 1.18181818],
[ 0.72727273, 1.27272727],
[ 0.44444444, 1.55555556],
[ 0.66666667, 1.33333333],
[ 0.66666667, 1.33333333],
[ nan, 2. ],
[ nan, 2. ]])
def _leaky():
problem = ecyglpki.Problem()
problem.add_rows(len(nutrition_target))
problem.add_cols(20)
# Configure rows/nutrients
for i, extrema in enumerate(nutrition_target):
problem.set_row_bnds(i+1, *extrema)
# Solve
int_opt_options = ecyglpki.IntOptControls()
int_opt_options.presolve = True # without this, you have to provide an LP relaxation basis
int_opt_options.msg_lev = 'no' # be quiet, no stdout
try:
problem.intopt(int_opt_options)
except ValueError:
pass
iterations = 50000
def very_leaky():
for _ in range(iterations):
_leaky()
def leaky():
for _ in range(iterations):
problem = ecyglpki.Problem()
problem.add_rows(len(nutrition_target))
problem.add_cols(20)
# Configure rows/nutrients
for i, extrema in enumerate(nutrition_target):
problem.set_row_bnds(i+1, *extrema)
# Solve
int_opt_options = ecyglpki.IntOptControls()
int_opt_options.presolve = True # without this, you have to provide an LP relaxation basis
int_opt_options.msg_lev = 'no' # be quiet, no stdout
try:
problem.intopt(int_opt_options)
except ValueError:
pass
def normal(): # for comparison
x=5
while True:
return np.ones(x)
import gc
from pympler import summary, refbrowser
sum1 = summary.summarize(gc.get_objects())
summary.print_(sum1)
very_leaky() # leaky leaks 1 leaky frame, very_leaky leaks $iterations _leaky frames
sum1 = summary.summarize(gc.get_objects())
summary.print_(sum1)
print('gc.collect')
gc.collect()
sum1 = summary.summarize(gc.get_objects())
summary.print_(sum1)
print('showing referrers tree')
frame = next(obj for obj in gc.get_objects() if type(obj).__name__ == 'frame' and obj.f_code.co_name in ('leaky', 'very_leaky'))
refbrowser.InteractiveBrowser(frame).main() # can take forever if iterations is set higher than 500
input()My CPython version: 3.5.2
pip freeze (not minimal; you probably only need pympler, numpy and ecyglpki):
apipkg==1.4
attrs==16.3.0
click==6.7
colored-traceback==0.2.2
coverage==4.3.1
coverage-pth==0.0.1
decorator==4.0.10
ecyglpki==0.2.0
execnet==1.4.1
networkx==1.11
numpy==1.11.3
pandas==0.19.2
plumbum==1.6.3
py==1.4.32
Pygments==2.1.3
Pympler==0.4.3
pyprof2calltree==1.4.0
pytest==3.0.5
pytest-asyncio==0.5.0
pytest-capturelog==0.7
pytest-cov==2.4.0
pytest-env==0.6.0
pytest-localserver==0.3.6
pytest-mock==1.5.0
pytest-xdist==1.15.0
python-dateutil==2.6.0
pytz==2016.10
six==1.10.0
tabulate==0.7.7
Werkzeug==0.11.15
When running the reproduce script below (which causes
intoptto raiseValueError),traceback,frame (codename: ecyglpki.Problem.intopt (ecyglpki.c:23524)),frame (codename: _leaky)objects are leaked. (The memory leaked is actually far greater than what the output reports below, it probably does not include C allocated memory).Examining the referrers tree,
frame(_leaky)is reffered to byframe(intopt)andtraceback(1).frame(intopt)is reffered to bytraceback(2); andtraceback(2)is referred to bytraceback(1)forming a cycle. Though, Python's GC supposedly handles cycles. If you run the reproduce script withiterationsset to5, you should get a Tk GUI allowing you to browse the referrer tree to the very_leaky frame.Output (from reproduce script below). First object listing is before running intopt (multiple times). Second is directly after it. Last listing is after running gc.collect:
To reproduce:
python script.pythe script below:My CPython version: 3.5.2
pip freeze (not minimal; you probably only need pympler, numpy and ecyglpki):