PySpark made easy

1. ```python
!pip install pyspark
```
Collecting pyspark
Downloading pyspark-2.2.0.post0.tar.gz (188.3MB)
Collecting py4j==0.10.4 (from pyspark)
Downloading py4j-0.10.4-py2.py3-none-any.whl (186kB)
Building wheels for collected packages: pyspark
Running setup.py bdist_wheel for pyspark: started
Running setup.py bdist_wheel for pyspark: finished with status
'done'
Stored in directory:
C:UsersDellAppDataLocalpipCachewheels5f0bb35cb16b15d28dcc32f8e
7ec91a044829642874bb7586f6e6cbe
Successfully built pyspark
Installing collected packages: py4j, pyspark
Successfully installed py4j-0.10.4 pyspark-2.2.0
```python
from pyspark import SparkContext,SparkConf
sc=SparkContext()
```
```python
import os
```
```python
os.getcwd()
```
'C:UsersDell'
```python
os.chdir('C:UsersDellDesktop')
```
```python
os.listdir()
```

2. ['desktop.ini',
'dump 2582017',
'Fusion Church.html',
'Fusion Church_files',
'iris.csv',
'KOG',
'NF22997109906610.ETicket.pdf',
'R Packages',
'Telegram.lnk',
'twitter_share.jpg',
'winutils.exe',
'~$avel Reimbursements.docx',
'~$thonajay.docx']
```python
#load data
data=sc.textFile('C:UsersDellDesktopiris.csv')
```
```python
type(data)
```
pyspark.rdd.RDD
```python
data.top(1)
```
['7.9,3.8,6.4,2,"virginica"']
```python
data.first()
```

3. '"Sepal.Length","Sepal.Width","Petal.Length","Petal.Width","Species"'
```python
from pyspark.sql import SparkSession
```
```python
spark= SparkSession.builder
.master("local")
.appName("Data Exploration")
.getOrCreate()
```
```python
#load data as Spark DataFrame
data2=spark.read.format("csv")
.option("header","true")
.option("mode","DROPMALFORMED")
.load('C:UsersDellDesktopiris.csv')
```
```python
type(data2)
```
pyspark.sql.dataframe.DataFrame
```python
data2.printSchema()
```
root
|-- Sepal.Length: string (nullable = true)
|-- Sepal.Width: string (nullable = true)
|-- Petal.Length: string (nullable = true)
|-- Petal.Width: string (nullable = true)
|-- Species: string (nullable = true)

4. ```python
data2.columns
```
['Sepal.Length', 'Sepal.Width', 'Petal.Length', 'Petal.Width',
'Species']
```python
data2.schema.names
```
['Sepal.Length', 'Sepal.Width', 'Petal.Length', 'Petal.Width',
'Species']
```python
newColumns=['Sepal_Length', 'Sepal_Width', 'Petal_Length', 'Petal_Width',
'Species']
```
```python
from functools import reduce
```
```python
data2 = reduce(lambda data2, idx:
data2.withColumnRenamed(oldColumns[idx], newColumns[idx]),
range(len(oldColumns)), data2)
data2.printSchema()
data2.show()
```
root
|-- Sepal_Length: string (nullable = true)
|-- Sepal_Width: string (nullable = true)
|-- Petal_Length: string (nullable = true)

5. |-- Petal_Width: string (nullable = true)
|-- Species: string (nullable = true)
+------------+-----------+------------+-----------+-------+
|Sepal_Length|Sepal_Width|Petal_Length|Petal_Width|Species|
+------------+-----------+------------+-----------+-------+
| 5.1| 3.5| 1.4| 0.2| setosa|
| 4.9| 3| 1.4| 0.2| setosa|
| 4.7| 3.2| 1.3| 0.2| setosa|
| 4.6| 3.1| 1.5| 0.2| setosa|
| 5| 3.6| 1.4| 0.2| setosa|
| 5.4| 3.9| 1.7| 0.4| setosa|
| 4.6| 3.4| 1.4| 0.3| setosa|
| 5| 3.4| 1.5| 0.2| setosa|
| 4.4| 2.9| 1.4| 0.2| setosa|
| 4.9| 3.1| 1.5| 0.1| setosa|
| 5.4| 3.7| 1.5| 0.2| setosa|
| 4.8| 3.4| 1.6| 0.2| setosa|
| 4.8| 3| 1.4| 0.1| setosa|
| 4.3| 3| 1.1| 0.1| setosa|
| 5.8| 4| 1.2| 0.2| setosa|
| 5.7| 4.4| 1.5| 0.4| setosa|
| 5.4| 3.9| 1.3| 0.4| setosa|
| 5.1| 3.5| 1.4| 0.3| setosa|
| 5.7| 3.8| 1.7| 0.3| setosa|
| 5.1| 3.8| 1.5| 0.3| setosa|
+------------+-----------+------------+-----------+-------+
only showing top 20 rows
```python
data2.dtypes
```
[('Sepal_Length', 'string'),
('Sepal_Width', 'string'),
('Petal_Length', 'string'),
('Petal_Width', 'string'),
('Species', 'string')]
```python
data3 = data2.select('Sepal_Length', 'Sepal_Width', 'Species')
data3.cache()
data3.count()
```

6. 150
```python
data3.show()
```
+------------+-----------+-------+
|Sepal_Length|Sepal_Width|Species|
+------------+-----------+-------+
| 5.1| 3.5| setosa|
| 4.9| 3| setosa|
| 4.7| 3.2| setosa|
| 4.6| 3.1| setosa|
| 5| 3.6| setosa|
| 5.4| 3.9| setosa|
| 4.6| 3.4| setosa|
| 5| 3.4| setosa|
| 4.4| 2.9| setosa|
| 4.9| 3.1| setosa|
| 5.4| 3.7| setosa|
| 4.8| 3.4| setosa|
| 4.8| 3| setosa|
| 4.3| 3| setosa|
| 5.8| 4| setosa|
| 5.7| 4.4| setosa|
| 5.4| 3.9| setosa|
| 5.1| 3.5| setosa|
| 5.7| 3.8| setosa|
| 5.1| 3.8| setosa|
+------------+-----------+-------+
only showing top 20 rows
```python
data3.limit(5)
```
DataFrame[Sepal_Length: string, Sepal_Width: string, Species: string]
```python

7. data3.limit(5).show()
```
+------------+-----------+-------+
|Sepal_Length|Sepal_Width|Species|
+------------+-----------+-------+
| 5.1| 3.5| setosa|
| 4.9| 3| setosa|
| 4.7| 3.2| setosa|
| 4.6| 3.1| setosa|
| 5| 3.6| setosa|
+------------+-----------+-------+
```python
data3.limit(5).limit(2).show()
```
+------------+-----------+-------+
|Sepal_Length|Sepal_Width|Species|
+------------+-----------+-------+
| 5.1| 3.5| setosa|
| 4.9| 3| setosa|
+------------+-----------+-------+
```python
data4=data2.selectExpr('CAST(Sepal_Length AS INT) AS Sepal_Length')
```
```python
data4
```
DataFrame[Sepal_Length: int]
```python
from pyspark.sql.functions import *
```
```python
data4.select('Sepal_Length').agg(mean('Sepal_Length')).show()

8. ```
+-----------------+
|avg(Sepal_Length)|
+-----------------+
|5.386666666666667|
+-----------------+
```python
data5=data2.selectExpr('CAST(Sepal_Length AS INT) AS
Sepal_Length','CAST(Petal_Width AS INT) AS Petal_Width','CAST(Sepal_Width
AS INT) AS Sepal_Width','CAST(Petal_Length AS INT) AS
Petal_Length','Species')
```
```python
data5
```
DataFrame[Sepal_Length: int, Petal_Width: int, Sepal_Width: int,
Petal_Length: int, Species: string]
```python
data5.columns
```
['Sepal_Length', 'Petal_Width', 'Sepal_Width', 'Petal_Length',
'Species']
```python
data5.select('Sepal_Length','Species').groupBy('Species').agg(mean("Sepal
_Length")).show()
```
+----------+-----------------+
| Species|avg(Sepal_Length)|
+----------+-----------------+
| virginica| 6.08|

9. |versicolor| 5.48|
| setosa| 4.6|
+----------+-----------------+
```python
#df =
data3.select(col('Sepal_Length'),dat.Sepal_Length.cast('float').alias('pr
ice'))
```