Install Kafka on Mac OS

To install Apache Kafka on Mac, Java is the only prerequisite. First we shall look into the installation steps of Java and then we shall setup Apache Kafka and run it on the Mac.

Install Java

  1. Open a browser and hit the url http://www.oracle.com/technetwork/java/javase/downloads/index.html.
  2. Click on JDK, check “Accept License Agreement” and download .dmg file for installation on Mac.
  3. Double click on the downloaded file and proceed with the installation steps.
  4. You have successfully installed Java. You may delete the .dmg file, as memory is costly on Mac.

You may verify the installation of Java on Mac, by running the following command on a Terminal.

$ java -version
Java version on Mac

Install Apache Kafka on Mac

1. Download the latest Apache Kafka from https://kafka.apache.org/downloads under Binary downloads.

Download Kafka Webpage

2. Click on any of the binary downloads, or choose a specific scala version if you have any dependency with scala in your development.

Kafka Mirror Download Link

3. Go with the recommended mirror site.

Save Kafka Package

4. Extract the contents.

Navigate to root of Apache Kafka folder and open a Terminal. Or Open a Terminal and navigate to the root directory of Apache Kafka.

apples-MacBook-Pro:kafka_2.12-1.0.0 Prasanth$ ls
bin config libs LICENSE logs NOTICE site-docs

Start Zookeeper

Apache Kafka depends on Zookeeper for cluster management. Hence, prior to starting Kafka, Zookeeper has to be started. There is no need to explicitly install Zookeeper, as it comes included with Apache Kafka.

From the root of Apache Kafka, run the following command to start Zookeeper :

~$ sh bin/zookeeper-server-start.sh config/zookeeper.properties

The zookeeper should be started with a similar following trace in the output.

apples-MacBook-Pro:kafka_2.12-1.0.0 Prasanth$ sh bin/zookeeper-server-start.sh config/zookeeper.properties 
[2017-12-31 15:18:36,556] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2017-12-31 15:18:36,560] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-12-31 15:18:36,560] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-12-31 15:18:36,560] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-12-31 15:18:36,560] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2017-12-31 15:18:36,582] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2017-12-31 15:18:36,582] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2017-12-31 15:18:36,650] INFO Server environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2017-12-31 15:18:36,650] INFO Server environment:host.name=192.168.0.104 (org.apache.zookeeper.server.ZooKeeperServer)

Start Apache Kafka Server

Open another Terminal and run the following command from the root of Apache Kafka to start Apache Kafka.

~$ sh bin/kafka-server-start.sh config/server.properties

Following should be the end of trace stating that Kafka server is started.

[2017-12-31 15:52:20,291] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@4690b489 (org.apache.zookeeper.ZooKeeper)
[2017-12-31 15:52:20,307] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2017-12-31 15:52:20,310] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2017-12-31 15:52:20,333] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2017-12-31 15:52:20,423] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x160abf917fa0000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2017-12-31 15:52:20,425] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2017-12-31 15:52:20,798] INFO Cluster ID = diRCJM2mQ4qI9KzjiiLRVA (kafka.server.KafkaServer)
[2017-12-31 15:52:20,820] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2017-12-31 15:52:20,866] INFO [ThrottledRequestReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-12-31 15:52:20,869] INFO [ThrottledRequestReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-12-31 15:52:20,870] INFO [ThrottledRequestReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-12-31 15:52:20,923] INFO Log directory '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)
[2017-12-31 15:52:20,943] INFO Loading logs. (kafka.log.LogManager)
[2017-12-31 15:52:20,973] INFO Logs loading complete in 29 ms. (kafka.log.LogManager)
[2017-12-31 15:52:21,155] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2017-12-31 15:52:21,159] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2017-12-31 15:52:21,535] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2017-12-31 15:52:21,538] INFO [SocketServer brokerId=0] Started 1 acceptor threads (kafka.network.SocketServer)
[2017-12-31 15:52:21,571] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-12-31 15:52:21,573] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-12-31 15:52:21,575] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-12-31 15:52:21,588] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2017-12-31 15:52:21,653] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-12-31 15:52:21,658] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-12-31 15:52:21,658] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-12-31 15:52:21,659] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-12-31 15:52:21,666] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2017-12-31 15:52:21,683] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2017-12-31 15:52:21,684] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2017-12-31 15:52:21,688] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 4 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2017-12-31 15:52:21,727] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2017-12-31 15:52:21,782] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2017-12-31 15:52:21,784] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2017-12-31 15:52:21,797] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2017-12-31 15:52:21,889] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-12-31 15:52:21,895] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2017-12-31 15:52:21,897] INFO Registered broker 0 at path /brokers/ids/0 with addresses: EndPoint(192.168.0.104,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2017-12-31 15:52:21,900] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2017-12-31 15:52:21,920] INFO Kafka version : 1.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2017-12-31 15:52:21,920] INFO Kafka commitId : aaa7af6d4a11b29d (org.apache.kafka.common.utils.AppInfoParser)
[2017-12-31 15:52:21,922] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Conclusion

In this Apache Kafka Tutorial, we have successfully learnt to install Apache Kafka on Mac and start Kafka along with Zookeeper.