Friday, December 15, 2017

Apache Kafka : Installing and Running on Windows

This notes will provide a step by step guide to run Apache Kafka on a windows OS. This guide will also provide instructions to setup Java & zookeeper. Apache kafka is a fast & scalable messaging queue, capable of handling real heavy loads in context of read & write. You can find more about kafka on kafka. Apache kafka requires a zookeeper instance running which is used for reliable distributed coordination. Please find more about Zookeeper on zookeeper.

Downloading the Required Files


Installation

1. Install & Setup Java Runtime Environment (JRE)

2. Installing & Running Zookeeper

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one. Otherwise Install ZooKeeper by following below steps.
  • Goto your zookeeper config directory. For me its C:\zookeeper-<version>\conf
  • Rename file “zoo_sample.cfg” to “zoo.cfg”
  • Open zoo.cfg in any text editor like notepad but I’ll prefer notepad++.
  • Find & edit dataDir=/tmp/zookeeper to dataDir=C:\zookeeper-<version>\data
  • Add entry in System Environment Variables as we did for java
  • Add in System Variables ZOOKEEPER_HOME = C:\zookeeper-<version>
  • Edit System Variable named “Path” add ;%ZOOKEEPER_HOME%\bin;
  • You can change the default zookeeper port in zoo.cfg file (Default port 2181).
  • Run zookeeper by opening a new cmd & type zkserver.
We can start zookeeper server from here or we can start from Kafka installation folder by using C:\kafka_2.11-0.9.0.0\config\zookeeper.properties.

3. Setting Up Kafka

  • Go to your Kafka config directory. For me its C:\kafka_2.11-0.9.0.0\config
  • Edit file “server.properties”
  • Find & edit line “log.dirs=/tmp/kafka-logs” to “log.dir= C:\kafka_2.11-0.9.0.0\kafka-logs”.
  • If your zookeeper is running on some other machine or cluster you can edit “zookeeper.connect=localhost:2181” to your custom IP & port. For this demo we are using same machine so no need to change. Also Kafka port & broker.id are configurable in this file. Leave other settings as it is.
  • Your Kafka will run on default port 9092 & connect to zookeeper’s default port which is 2181. 

4. Running Kafka Server

Please ensure that your zookeeper is up & running before starting Kafka server.
  • Go to your kafka installation directory C:\kafka_2.11-0.9.0.0\
  • Open command line in your Kafka installation folder
  • Launch Zookeeper with .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
  • Open a second command line in your Kafka installation folder
  • Launch single Kafka broker: .\bin\windows\kafka-server-start.bat .\config\server.properties
Now your Kafka is up & running, so now you can create topics & produce or consume data from java/scala code & command prompt also.

5. Creating Kafka Topic

  • Now create a topic with name “test” & replication factor 1 as we have only one Kafka server running. If you have a cluster with more than 1 Kafka servers running, you can increase the replication-factor accordingly which will increase the data availability & act like a fault-tolerant system.
  • Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows
  • Type following command and hit enter
  • kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

6. Creating a producer & consumer to test server

  • Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows
  • To start a producer type command “kafka-console-producer.bat --broker-list localhost:9092 --topic test”.
  • Again open a new command prompt in the same location as C:\kafka_2.11-0.9.0.0\bin\windows
  • Now start a consumer by typing command “kafka-console-consumer.bat --zookeeper localhost:2181 --topic test”.
Now you will have two command prompts like image below...

  • Now type anything in the producer command prompt & press enter and you should be able to see the message in the other consumer command prompt.
  • If you are able to push & see your messages in consumer side, your Kafka setup is ready.


0 comments:

Post a Comment

Contact

Get in touch with me


Adress/Street

Bangalore, India