Spark Installation Windows - WordPress.comSpark Installation Windows - AaBbccDc Normal Microsoft...
Transcript of Spark Installation Windows - WordPress.comSpark Installation Windows - AaBbccDc Normal Microsoft...
![Page 1: Spark Installation Windows - WordPress.comSpark Installation Windows - AaBbccDc Normal Microsoft Word AaBbCcDc No Spaci.. AaBbCc Heading 2 Styles Title 4aBbCc. Subtitle AaBbccDt Subtle](https://reader034.fdocuments.us/reader034/viewer/2022042210/5eae9895d6b12e55c15fabee/html5/thumbnails/1.jpg)
Step 1: Download & Untar SPARK
Download the version 1.0.2 of spark
Untar the downloaded file to any location (say C:\spark-‐1.0.2)
Step 2: Download SBT msi (needed for Windows)
Download sbt.MSI & execute it.
![Page 2: Spark Installation Windows - WordPress.comSpark Installation Windows - AaBbccDc Normal Microsoft Word AaBbCcDc No Spaci.. AaBbCc Heading 2 Styles Title 4aBbCc. Subtitle AaBbccDt Subtle](https://reader034.fdocuments.us/reader034/viewer/2022042210/5eae9895d6b12e55c15fabee/html5/thumbnails/2.jpg)
![Page 3: Spark Installation Windows - WordPress.comSpark Installation Windows - AaBbccDc Normal Microsoft Word AaBbCcDc No Spaci.. AaBbCc Heading 2 Styles Title 4aBbCc. Subtitle AaBbccDt Subtle](https://reader034.fdocuments.us/reader034/viewer/2022042210/5eae9895d6b12e55c15fabee/html5/thumbnails/3.jpg)
You may need to restart the machine so that command line can identify the sbt command
![Page 4: Spark Installation Windows - WordPress.comSpark Installation Windows - AaBbccDc Normal Microsoft Word AaBbCcDc No Spaci.. AaBbCc Heading 2 Styles Title 4aBbCc. Subtitle AaBbccDt Subtle](https://reader034.fdocuments.us/reader034/viewer/2022042210/5eae9895d6b12e55c15fabee/html5/thumbnails/4.jpg)
Step 3: Package Spark using SBT
C:\spark-‐1.0.2>sbt assembly
Note: This step takes enormous amount of time. Please be patient
Step 4: Download SCALA
Spark 1.0.2 needs Scala 2.10. This is extremely important to note. And you can read the README.MD file in the SPARK folder to find the correct scala version needed for your spark.
Download and unzip the scala to any location (say C:\ scala-‐2.10.1)
Set SCALA_HOME environment variable & set the PATH variable to the bin directory of scala
Verify the scala version (and thus the download)
Step 5: Start the spark shell
C:\spark-‐1.0.2\bin>spark-‐shell
![Page 5: Spark Installation Windows - WordPress.comSpark Installation Windows - AaBbccDc Normal Microsoft Word AaBbCcDc No Spaci.. AaBbCc Heading 2 Styles Title 4aBbCc. Subtitle AaBbccDt Subtle](https://reader034.fdocuments.us/reader034/viewer/2022042210/5eae9895d6b12e55c15fabee/html5/thumbnails/5.jpg)
Sample program in SPARK
1) Create a data set of 1…10000 integers scala> val data = 1 to 10000
2) Use Spark Context to create an RDD [Resilient Distributed Dateset] from that data
scala> val distData = sc.parallelize(data)
3) Perform a filter mechanism on that data scala> distData.filter(_ < 10).collect()