out of memory

[Squitoense]

Unfortunately the program exits with an out of memory error:
pandas.errors.ParserError: Error tokenizing data. C error: out of memory
My machine has 16 GB of RAM, hoping this is sufficient for the script to run.
Is there anything I can do to make it run? I am not as proficient in python as you.

[ltirrell]

could you tell me the command you ran to get that error, and paste the entire terminal output so I can debug it further?

Unfortunately, combining all the CITES data into a single file uses a lot of memory. I'm planning to refactor and improve this repo next month to get around the issue next month.

I've also ran into memory usage on a laptop with 16 GB RAM when I have a lot of apps open (Chrome, Slack, VS Code, etc). One option to try is to restart your computer, and just run the pycites command with no other programs running to see if that helps.

Also, starting with v0.1.4 and going forward, I upload the pre-created CSV of all the data as part of the release. You can find the Trade_database_download_v2021.1.csv.gz on the release page here. This is the file created by pycites.get_data(), and contains all the CITES data so you can use it for any analysis you want to do

[Squitoense]

Thanks for your quick answer. I will check out the *.csv.gz as this is all I need for my database. I just ran the following command:

import pycites

pycites.get_data()
df = pycites.load_data()

The download of the data works fine, the error is in reading the csv:

37%|███▋ | 17/46 [00:24<00:42, 1.46s/files]C:\Users\username\AppData\Local\ltirrell\pycites\Cache\Trade_database_download_v2021.1\Trade_database_download_v2021.1\trade_db_25.csv

37%|███▋ | 17/46 [00:25<00:42, 1.48s/files]
Traceback (most recent call last):
File "F:\python\speciesdb\cites_trade_data\pycites_module.py", line 3, in
pycites.get_data()
File "C:\Users\username\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pycites\cites_download.py", line 180, in get_data
df = combine_csv((zip_file.parent) / zip_file.stem, cleanup)
File "C:\Users\username\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pycites\cites_download.py", line 82, in combine_csv
d = pd.read_csv(f, low_memory=False)
File "C:\Users\username\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\util_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "C:\Users\username\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\io\parsers\readers.py", line 586, in read_csv
return _read(filepath_or_buffer, kwds)
File "C:\Users\username\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\io\parsers\readers.py", line 488, in _read
return parser.read(nrows)
File "C:\Users\username\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\io\parsers\readers.py", line 1047, in read
index, columns, col_dict = self._engine.read(nrows)
File "C:\Users\username\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 228, in read
data = self._reader.read(nrows)
File "pandas_libs\parsers.pyx", line 783, in pandas._libs.parsers.TextReader.read
File "pandas_libs\parsers.pyx", line 872, in pandas._libs.parsers.TextReader._read_rows
File "pandas_libs\parsers.pyx", line 1925, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: out of memory
[Finished in 52.4s]