Install Latest Apache Spark on Mac OS

Following is a detailed step by step process to install latest Apache Spark on Mac OS. We shall first install the dependencies : Java and Scala. To install these programming languages and framework, we take help of Homebrew and xcode-select.

Install Spark on Mac OS - Apache Spark Tutorial - www.tutorialkart.com
Install Spark on Mac OS

Step 1 : Install Homebrew

Open Terminal. Run the following command in Terminal :

$ /usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”

Enter the password if asked and continue.

==> Cleaning up /Library/Caches/Homebrew...
==> Migrating /Library/Caches/Homebrew to /Users/apple/Library/Caches/Homebrew..
==> Deleting /Library/Caches/Homebrew...
Already up-to-date.
==> Installation successful!
 
==> Homebrew has enabled anonymous aggregate user behaviour analytics.
Read the analytics documentation (and how to opt-out) here:
  http://docs.brew.sh/Analytics.html
 
==> Next steps:
- Run `brew help` to get started
- Further documentation: 
    http://docs.brew.sh

Step 2 : Install xcode-select

To install Java, Scala and Apache Spark through command line interface in Terminal, we shall install xcode-select. Enter and run the following command in Terminal :

$ xcode-select –install

Step 3 : Install Java

To install Java through command line, enter and run the following command in the Terminal :

$ brew cask install java
If your Java application still asks for JRE installation, you might need
to reboot or logout/login.
 
Installing this Cask means you have AGREED to the Oracle Binary Code
License Agreement for Java SE at
 
  https://www.oracle.com/technetwork/java/javase/terms/license/index.html
 
==> Satisfying dependencies
==> Downloading http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda
######################################################################## 100.0%
==> Verifying checksum for Cask java
==> Installing Cask java
==> Running installer for java; your password may be necessary.
==> Package installers may write to any location; options such as --appdir are i
Password:
==> installer: Package name is JDK 8 Update 144
==> installer: Upgrading at base path /
==> installer: The upgrade was successful.
? java was successfully installed!

Step 4 : Install Scala

To install Scala through command line, enter and run the following command in Terminal :

$ brew install scala
apples-MBP:~ Prasanth$ brew install scala
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
grafana                    idris                      passenger
 
==> Using the sandbox
==> Downloading https://downloads.lightbend.com/scala/2.12.3/scala-2.12.3.tgz
######################################################################## 100.0%
==> Downloading https://raw.githubusercontent.com/scala/scala-tool-support/0a217
######################################################################## 100.0%
==> Caveats
To use with IntelliJ, set the Scala home to:
  /usr/local/opt/scala/idea
 
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d
==> Summary
? /usr/local/Cellar/scala/2.12.3: 44 files, 20.1MB, built in 1 minute 1 second

Step 5 : Install Spark

To install Apache Spark through command line, enter and run the following command in the Terminal :

$ brew install apache-spark
apples-MBP:~ Prasanth$ brew install apache-spark
==> Using the sandbox
==> Downloading https://www.apache.org/dyn/closer.lua?path=spark/spark-2.2.0/spa
==> Best Mirror http://www-eu.apache.org/dist/spark/spark-2.2.0/spark-2.2.0-bin-
######################################################################## 100.0%
? /usr/local/Cellar/apache-spark/2.2.0: 1,318 files, 221.5MB, built in 12 minutes 8 seconds

Step 6 : Verifying installation

To verify if the installation is successful, run the spark using the following command in Terminal :

$ spark-shell
apples-MBP:~ Prasanth$ spark-shell
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/08/01 21:52:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/01 21:52:58 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/08/01 21:52:58 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
17/08/01 21:52:59 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.1.101:4040
Spark context available as 'sc' (master = local[*], app id = local-1501604572582).
Spark session available as 'spark'.
Welcome to
      __              __
     / _/_  _ __/ /_
    \ \/ _ \/ _ `/ _/  '_/
   /_/ ._/\,// //\\   version 2.2.0
      /_/
        
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
Type in expressions to have them evaluated.
Type :help for more information.
 
scala>

We have successfully installed Apache Spark on Mac OS.

The installation directory would be /usr/local/Cellar/apache-spark/.

Conclusion

In this Apache Spark Tutorial, we have learnt to install latest Apache Spark on Mac OS .