Install Latest Apache Spark on Mac OS
Following is a detailed step by step process to install latest Apache Spark on Mac OS. We shall first install the dependencies : Java and Scala. To install these programming languages and framework, we take help of Homebrew and xcode-select.
Step 1 : Install Homebrew
Open Terminal. Run the following command in Terminal :
$ /usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”
Enter the password if asked and continue.
==> Cleaning up /Library/Caches/Homebrew...
==> Migrating /Library/Caches/Homebrew to /Users/apple/Library/Caches/Homebrew..
==> Deleting /Library/Caches/Homebrew...
Already up-to-date.
==> Installation successful!
==> Homebrew has enabled anonymous aggregate user behaviour analytics.
Read the analytics documentation (and how to opt-out) here:
http://docs.brew.sh/Analytics.html
==> Next steps:
- Run `brew help` to get started
- Further documentation:
http://docs.brew.sh Step 2 : Install xcode-select
To install Java, Scala and Apache Spark through command line interface in Terminal, we shall install xcode-select. Enter and run the following command in Terminal :
$ xcode-select –install
Step 3 : Install Java
To install Java through command line, enter and run the following command in the Terminal :
$ brew cask install java
If your Java application still asks for JRE installation, you might need to reboot or logout/login. Installing this Cask means you have AGREED to the Oracle Binary Code License Agreement for Java SE at https://www.oracle.com/technetwork/java/javase/terms/license/index.html ==> Satisfying dependencies ==> Downloading http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda ######################################################################## 100.0% ==> Verifying checksum for Cask java ==> Installing Cask java ==> Running installer for java; your password may be necessary. ==> Package installers may write to any location; options such as --appdir are i Password: ==> installer: Package name is JDK 8 Update 144 ==> installer: Upgrading at base path / ==> installer: The upgrade was successful. ? java was successfully installed!
Step 4 : Install Scala
To install Scala through command line, enter and run the following command in Terminal :
$ brew install scala
apples-MBP:~ Prasanth$ brew install scala Updating Homebrew... ==> Auto-updated Homebrew! Updated 1 tap (homebrew/core). ==> Updated Formulae grafana idris passenger ==> Using the sandbox ==> Downloading https://downloads.lightbend.com/scala/2.12.3/scala-2.12.3.tgz ######################################################################## 100.0% ==> Downloading https://raw.githubusercontent.com/scala/scala-tool-support/0a217 ######################################################################## 100.0% ==> Caveats To use with IntelliJ, set the Scala home to: /usr/local/opt/scala/idea Bash completion has been installed to: /usr/local/etc/bash_completion.d ==> Summary ? /usr/local/Cellar/scala/2.12.3: 44 files, 20.1MB, built in 1 minute 1 second
Step 5 : Install Spark
To install Apache Spark through command line, enter and run the following command in the Terminal :
$ brew install apache-spark
apples-MBP:~ Prasanth$ brew install apache-spark ==> Using the sandbox ==> Downloading https://www.apache.org/dyn/closer.lua?path=spark/spark-2.2.0/spa ==> Best Mirror http://www-eu.apache.org/dist/spark/spark-2.2.0/spark-2.2.0-bin- ######################################################################## 100.0% ? /usr/local/Cellar/apache-spark/2.2.0: 1,318 files, 221.5MB, built in 12 minutes 8 seconds
Step 6 : Verifying installation
To verify if the installation is successful, run the spark using the following command in Terminal :
$ spark-shell
apples-MBP:~ Prasanth$ spark-shell
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/08/01 21:52:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/01 21:52:58 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/08/01 21:52:58 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
17/08/01 21:52:59 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.1.101:4040
Spark context available as 'sc' (master = local[*], app id = local-1501604572582).
Spark session available as 'spark'.
Welcome to
__ __
/ _/_ _ __/ /_
\ \/ _ \/ _ `/ _/ '_/
/_/ ._/\,// //\\ version 2.2.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
Type in expressions to have them evaluated.
Type :help for more information.
scala> We have successfully installed Apache Spark on Mac OS.
The installation directory would be /usr/local/Cellar/apache-spark/.
Conclusion
In this Apache Spark Tutorial, we have learnt to install latest Apache Spark on Mac OS .