Big Data

List of templates for the processing of big data (out of memory). PySpark ...

A note on the notebook rendering

Each notebook has two versions (all python scripts are unaffected by this):

One where all the markdown comments are rendered in black& white. These are placed in the folder named GitHub_MD_rendering where MD stands for MarkDown.
One where all the markdown comments are rendered in coloured.

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
GitHub_MD_rendering		GitHub_MD_rendering
.gitignore		.gitignore
Batch Processing 22GB of Transaction Data with Pandas.ipynb		Batch Processing 22GB of Transaction Data with Pandas.ipynb
Classification with SGD - minibatch - large dataset.ipynb		Classification with SGD - minibatch - large dataset.ipynb
How to compute out-of-memory statistics.ipynb		How to compute out-of-memory statistics.ipynb
README.md		README.md
Train 1 billion samples in 5 minutes on your laptop using Vaex and Scikit-Learn.ipynb		Train 1 billion samples in 5 minutes on your laptop using Vaex and Scikit-Learn.ipynb
downloadLargeDataset.py		downloadLargeDataset.py