Home » Posts tagged 'Kafka Producer'

Tag Archives: Kafka Producer

Install Kafka on Windows

This post is a step-by-step guide to install and run Apache Kafka on Windows.

Prerequisite

The only prerequisite for this setup is JRE.

Install Java (Skip if you already have it)

  1. Download Java 8 from here. (Java 8 is recommended by Apache Kafka)
  2. Run the installer and follow the instructions on Installation wizard.
  3. Please note/copy the Destination Folder location.
  4. Go to Control Panel -> System -> Advanced system settings -> Environment Variables.
  5. Create a new user variable named JAVA_HOME and paste the path copied from step 3 to the variable value and click OK.
  6. Now edit PATH variable in User variables and add “%JAVA_HOME%\bin” at the end of the variable value. If it’s an older windows version, then add “;%JAVA_HOME%\bin;” at the end of the text. If PATH variable doesn’t exist create it with the value “%JAVA_HOME%\bin”.
  7. Open command prompt and type “java -version” to validate the installation.
  8. If you get the following output in your command prompt you’re good to go:
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)

Steps to install Kafka on windows:

1) Download and extract Apache Kafka from here.

Note: At the time of writing this post the current stable version is 2.4.0. If you’re using Scala for development then make sure to select the Kafka version corresponding to your Scala version.

2) Go to the folder where you’ve extracted Kafka, open the file server.properties in the config folder and then edit the line log.dirs=/tmp/kafka-logs to log.dirs=C:\path_where_kafka_is_extracted\kafka-logs.

Note: If you won’t change log.dirs value, you’ll keep getting the following error:

java.util.NoSuchElementException: key not found: /tmp/kafka-logs

3) Start Zookeeper

Kafka uses ZooKeeper so before starting Kafka you have to make sure that ZooKeeper is up and running. We’ll be running the single node ZooKeeper instance packaged with Kafka. 

Go to your Kafka installation directory and open command prompt and start Zookeeper using the following command:

> bin\windows\zookeeper-server-start.bat config\zookeeper.properties

Once zookeeper is started you should see this in the command prompt:

INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

4) Start Kafka Server

You need to open another command prompt in the Kafka installation directory and run the following command:

> bin\windows\kafka-server-start.bat \config\server.properties

Once the Kafka server is started you should see something like this in the command prompt:

INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

5) Validate the Kafka setup

Create a Kafka topic:

Open a command prompt in the Kafka installation directory and run the following command:

> bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic

The above command will create a topic “testTopic” with 1 partition and 1 replication factor.

Produce messages to the created topic:

In the same command prompt start a kafka-console-producer and produce some messages:

> bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic testTopic
>Hello
>How are you?

Consume the messages produced by the kafka-console-producer:

Open another command prompt in the Kafka installation directory and start a kafka-console-consumer which subscribes to testTopic:

> bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic testTopic --from-beginning
Hello
How are you?

The messages which are produced by the kafka-console-producer will be visible on the kafka-console-consumer console.

Congratulations on completing a single broker Kafka setup on windows!

Install Kafka on macOS

I really like Homebrew to install stuff on macOS. So in this tutorial I’ll be using Hombrew to install Apache Kafka on macOS.

Install Homebrew:

Open Terminal, paste the following command and then press Return key:

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

The above command will install Homebrew. Once it is installed, follow the next steps:

1)  Install Java8:

After Homebrew is installed, you can just paste the following commands in Terminal and press Return to install java on macOS:

$ brew tap adoptopenjdk/openjdk
$ brew cask install adoptopenjdk8

The above commands will install openjdk 8. In order to validate the java installation, execute the following command in Terminal:

$ java -version

If you get the following output, it means that java is installed correctly:

openjdk version "1.8.0_212"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_212-b04)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.212-b04, mixed mode)

2) Install Kafka:

To install Kafka on macOS: open Terminal, paste the following command and then press Return:

$ brew install kafka

The above command will install Kafka along with Zookeeper.

Start Zookeeper:

To start ZooKeeper, paste the following command in the Terminal and press Return:

$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

Once ZooKeeper is started you’ll see something like this in the console:

INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

Start Kafka:

To start Kafka, paste the following command in another Terminal and press Return:

$ kafka-server-start /usr/local/etc/kafka/server.properties

Once Kafka is started you’ll see something like this in the console:

INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Create a Kafka topic:

Open another Terminal and execute the following command to create a topic testTopic with 1 partition and 1 replication factor:

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic

Produce messages to the created topic:

In the same Terminal prompt, start a kafka-console-producer and produce some messages:

$ kafka-console-producer --broker-list localhost:9092 --topic testTopic
>Hello
>How are you?

Consume the messages produced by the producer:

Open another Terminal and start a kafka-console-consumer which subscribes to testTopic:

$ kafka-console-consumer --bootstrap-server localhost:9092 --topic testTopic --from-beginning
Hello
How are you?

The messages which are produced by the kafka-console-producer will be visible on the kafka-console-consumer console.

Congratulations on setting up a single broker Kafka setup on macOS!