Install Apache Kafka on Ubuntu

To install Apache Kafka on Ubuntu, Java is the only prerequisite. First we shall look into the installation steps of Java and then we shall setup Apache Kafka and run it on the Ubuntu.

Install Apache Kafka on Ubuntu - Setup Apache Kafka on Ubuntu - Apache Kafka Tutorial - www.tutorialkart.com
Install Apache Kafka in Ubuntu

Install Java

Open a terminal and run the following command :

$ sudo apt-get install default-jdk

To verify the installation of java, run the following command in the terminal :

$ java -version

The output should show the version of java installed, similar to below verbose :

kafkauser@tutorialkart:~$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

Setup Apache Kafka

Download the latest Apache Kafka from https://kafka.apache.org/downloads under Binary downloads.

Extract the .tgz file.

Navigate to root of Apache Kafka folder and open a Terminal. Or Open a Terminal and navigate to the root directory of Apache Kafka.

kafkauser@tutorialkart:/home/kafkauser/kafka/kafka_2.11-0.11.0.0$ ls
bin config libs LICENSE logs NOTICE site-docs

Start Zookeeper

Apache Kafka depends on Zookeeper for cluster management. Hence, prior to starting Kafka, Zookeeper has to be started. There is no need to explicitly install Zookeeper, as it comes included with Apache Kafka.

From the root of Apache Kafka, run the following command to start Zookeeper :

~$ bin/zookeeper-server-start.sh config/zookeeper.properties

The zookeeper should be started with a similar following trace in the output.

kafkauser@tutorialkart:/home/kafkauser/kafka/kafka_2.11-0.11.0.0~$ bin/zookeeper-server-start.sh config/zookeeper.properties
[2017-08-14 10:34:02,974] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2017-08-14 10:34:03,035] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-08-14 10:34:03,036] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-08-14 10:34:03,036] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-08-14 10:34:03,036] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2017-08-14 10:34:03,061] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2017-08-14 10:34:03,083] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2017-08-14 10:34:03,131] INFO Server environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT (org.apache.zookeeper.server.ZooKeeperServer)

Start Apache Kafka Server

Open another Terminal and run the following command from the root of Apache Kafka to start Apache Kafka.

~$ bin/kafka-server-start.sh config/server.properties

Following should be the end of trace stating that Kafka server is started.

[2017-08-14 10:38:11,070] INFO Cluster ID = xp4xG4pbSEGnLXXW5hB8qA (kafka.server.KafkaServer)
[2017-08-14 10:38:11,076] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2017-08-14 10:38:11,198] INFO [ThrottledRequestReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-08-14 10:38:11,199] INFO [ThrottledRequestReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-08-14 10:38:11,201] INFO [ThrottledRequestReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-08-14 10:38:11,353] INFO Log directory '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)
[2017-08-14 10:38:11,381] INFO Loading logs. (kafka.log.LogManager)
[2017-08-14 10:38:11,391] INFO Logs loading complete in 10 ms. (kafka.log.LogManager)
[2017-08-14 10:38:11,460] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2017-08-14 10:38:11,463] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2017-08-14 10:38:11,552] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2017-08-14 10:38:11,555] INFO [Socket Server on Broker 0], Started 1 acceptor threads (kafka.network.SocketServer)
[2017-08-14 10:38:11,588] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-14 10:38:11,590] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-14 10:38:11,592] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-14 10:38:11,691] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-14 10:38:11,696] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-14 10:38:11,698] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-14 10:38:11,700] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-08-14 10:38:11,732] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2017-08-14 10:38:11,762] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2017-08-14 10:38:11,764] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2017-08-14 10:38:11,767] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 4 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2017-08-14 10:38:11,894] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2017-08-14 10:38:11,990] INFO [Transaction Coordinator 0]: Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2017-08-14 10:38:11,995] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2017-08-14 10:38:11,995] INFO [Transaction Coordinator 0]: Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2017-08-14 10:38:12,081] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2017-08-14 10:38:12,179] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-08-14 10:38:12,192] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2017-08-14 10:38:12,194] INFO Registered broker 0 at path /brokers/ids/0 with addresses: EndPoint(arjun-VPCEH26EN,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2017-08-14 10:38:12,195] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2017-08-14 10:38:12,259] INFO Kafka version : 0.11.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2017-08-14 10:38:12,259] INFO Kafka commitId : cb8625948210849f (org.apache.kafka.common.utils.AppInfoParser)
[2017-08-14 10:38:12,261] INFO [Kafka Server 0], started (kafka.server.KafkaServer)

Conclusion

In this Apache Kafka Tutorial, we have successfully learnt to install Apache Kafka on Ubuntu and start Kafka along with Zookeeper.