With 20 million rows, pandas is quite slow for reading in data and manipulating it. After a quick assessment of modin and vaex, vaex seems like an easy to use and fast solution. modin was a bit slow for my use case. dask is another option, but based on benchmarks posted online, it also seems like it won't lead to much speed up over raw pandas (though lazy evaluation would probably lead to less swapping).
With 20 million rows, pandas is quite slow for reading in data and manipulating it. After a quick assessment of
modinandvaex,vaexseems like an easy to use and fast solution.modinwas a bit slow for my use case.daskis another option, but based on benchmarks posted online, it also seems like it won't lead to much speed up over raw pandas (though lazy evaluation would probably lead to less swapping).