Basics

This file is mainly to understand the basic concepts in data science programming languagues like Python,R,Dask,PySpark etc.,

Setting Up Spark Environment

Note: If you dont have Java Environment set up in your machine download JDK(13.0) from https://www.oracle.com/java/technologies/javase-downloads.html and set the path in environmental variables

1. Download spark-2.4.5-bin-hadoop2.7 from https://spark.apache.org/downloads.html
2. Download Hadoop 2.7 WinUtils from https://dzone.com/articles/working-on-apache-spark-on-windows
3. Set environmental variables for both spark and hadoop as SPARK_HOME and Hadoop_HOME and give the paths where you installed
4. SPARK_HOME:
5. HADOOP_HOME:
6. Set this environmental variable ---> PYSPARK_DRIVER_PYTHON
7. Add PATH VARIABLE:
8. Set this environmental variable PYSPARK_DRIVER_PYTHON_OPTS
9. To start pyspark after setting environmental variables type the below command in CMD pyspark --master local[2] or I opened Jupter notebook and did the following commands By the above command you are starting pyspark on 2 cores
10. Once you open Jupyter note book

or try this dummy commands to check spark working or not spark = SparkSession.builder.appName('abc').getOrCreate() df=spark.read.csv('wx_pv_corr.csv',header=True)