ResourceWarning: Unclosed file... messages when deserializing a model.

[briantani]

The load_obj in the pickle_utils.py file makes lots of use of this pattern:
dill.load(open(os.path.join(dir_name, file), 'rb'))
And since dill does not close the file handler passed to the load method, it causes a Resource Warning for each of the parts of the model being loaded. In the following case, it caused 2856 of those warnings flooding the logs.

c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4762240Z 
2022-08-20T16:56:02.4762597Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4.type'>
2022-08-20T16:56:02.4762842Z 
2022-08-20T16:56:02.4763280Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4764328Z 
2022-08-20T16:56:02.4764749Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\df_dropna__value__.type'>
2022-08-20T16:56:02.4765023Z 
2022-08-20T16:56:02.4765471Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4765803Z 
2022-08-20T16:56:02.4766176Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\df__value__.type'>
2022-08-20T16:56:02.4766440Z 
2022-08-20T16:56:02.4766859Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4767206Z 
2022-08-20T16:56:02.4767571Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\features_df__value__.type'>
2022-08-20T16:56:02.4767866Z 
2022-08-20T16:56:02.4768299Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4768626Z 
2022-08-20T16:56:02.4769008Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\fitted_df__value__.type'>
2022-08-20T16:56:02.4769279Z 
2022-08-20T16:56:02.4769714Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4770043Z 
2022-08-20T16:56:02.4770431Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\fs_components_df__value__.type'>
2022-08-20T16:56:02.4770714Z 
2022-08-20T16:56:02.4771148Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4771476Z 
2022-08-20T16:56:02.4771863Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\ml_model_summary__value__.type'>
2022-08-20T16:56:02.4772143Z 
2022-08-20T16:56:02.4772588Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4772917Z 
2022-08-20T16:56:02.4773290Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\uncertainty_model__value__.type'>
2022-08-20T16:56:02.4773589Z 
2022-08-20T16:56:02.4774004Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4774354Z 
2022-08-20T16:56:02.4774721Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\x_design_info__value__.type'>
2022-08-20T16:56:02.4775010Z 
2022-08-20T16:56:02.4775425Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:313: ResourceWarning:
2022-08-20T16:56:02.4776102Z 
2022-08-20T16:56:02.4776491Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\x_mat__value__.type'>
2022-08-20T16:56:02.4776755Z 
2022-08-20T16:56:02.4777188Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:376: ResourceWarning:
2022-08-20T16:56:02.4777512Z 
2022-08-20T16:56:02.4777900Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\adjust_anomalous_info__key__.pkl'>
2022-08-20T16:56:02.4778437Z 
2022-08-20T16:56:02.4778891Z c:\windows\serviceprofiles\networkservice\.conda\envs\aml_test_38\lib\site-packages\greykite\framework\templates\pickle_utils.py:395: ResourceWarning:
2022-08-20T16:56:02.4779216Z 
2022-08-20T16:56:02.4779615Z unclosed file <_io.BufferedReader name='C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp\\tmp55hy74nb\\tmp7q08pus4\\adjust_anomalous_info__value__.pkl'>
...

We were using graykite 0.2.0, but I see the issue is still present in version 0.4.0.

If instead we had a simple read_pickle or dill_load function that would handle closing the file:

def read_pickle(path, mode):
    """Loads the pickled files and closes the file handle.

    Parameters
    ----------
    path : `str`
        The path to the pickled file.
    mode : `str`
        The mode of the open function.
    """
    with open(path, mode) as file:
        data = dill.load(file)
    return data

It would be just a question of changing the calls to dill.load(open(...)) to read_pickle(...). Are there any reason not to use this pattern?

[KaixuYang]

Thanks for raising this issue! We will look into this issue. On the first look it seems this is with the Windows system.