Would you like to understand the basic concept of Apache Kafka? See our previous blog on the basic concept of Apache Kafka.

Download

Apache Kafka

You can download Kafka version 3.5.0 from a link  https://kafka.apache.org/downloads. 

Zookeeper

You can download Zookeeper version 3.8.1 from a link  https://kafka.apache.org/downloads.

Important: For Zookeeper, need to download the .bin-tar file. For Kafka, need to download the Scala 2.13 tar file.

Installation

• Move Kafka and zookeeper tar to the Kafka directory.
• Extract the kafka and zookeeper tar files using commands like (tar -xf kafka_2.13_3.4.1/apache-zookeeper-3.8.1-bin

Configuration

Apache Kafka uses key-value pairs in the property file format for configuration. These values can be passed either from a file or programmatically.

Zookeeper and Kafka Config

Before starting the Kafka server need to start the Zookeeper server first.

• For Windows
  • Move the Kafka folder close to the root directory like(D:/kafka).
  • Make sure there is no space or “-” character to your directory path name.
  • Open the cmd or power shell as Administrator.

Write the following command to start the zookeeper server.

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Write the following command to start the Kafka server.

.\bin\windows\kafka-server-start.bat .\config\server.properties 

• For Linux
  • Copy the zoo_sample.cfg file to zoo.cfg on \apache-zookeeper-3.8.1-bin\conf path.
  • Edit zoo.cfg file to remove the comment from maxClientCnxns=60 property and add one more property like 4lw.commands.whitelist=* to run 4 line word commands in cmd to know server-related activities.
  • Open the Command prompt using the “cmd” command and write the following command to start the start zookeeper server.

bin/zk-Server.sh for and write command bin/kafka-server-start.sh config/server.properties.

• To validate to start the server using the command. We need to write the following command to know whether the server is started or not. This command works only if we configure the 4lw property to zoo.cfg file.

 echo stat | nc localhost 2181 

Topic Configuration

• For creating a new Kafka Topic, open a separate command prompt window:
• Use the –create command to create a topic.

Important:
1. Instead of ‘.bat’, use ‘.sh’ while creating topics on Linux.
2. The replication factor can never be greater than the number of available brokers.
3. If you are using an old Kafka version (<2.0 version), use –zookeeper localhost:2181 in commands.
4. If you are using the recent Kafka versions (>2.0 version), use the command with –bootstrap-server localhost:9092.

The command consists of attributes like Create, bootstrap-servers, localhost:9092, Replication-factor, Partitions:

• Create: It is a command for creating a new Kafka topic.
• Partitions: The newly created Topics can be divided and stored in one or more Partitions to enable the balancing of messages or loads.
• Replication Factor: The Replication Factor defines the number of copies or replicas of a Topic across the Kafka Cluster. If you give the Replication Factor as 1, you are making one copy or replication of your newly created Topic. Similarly, when you provide the Replication Factor as 2, you make two copies for the respective Topic. This Replication feature ensures the Kafka Server is highly fault-tolerant. Since it replicates and spreads the topics across other Kafka Servers, if one of the servers fails, the topic will be available on other servers.
• Bootstrap Server: It provides the initial hosts that act as the starting point for a Kafka client to discover the full set of live servers in the cluster.
• Localhost:9092: It specified “localhost:9092” as the bootstrap server to the Kafka cluster listens to the 9092 port.
• Run a command to create a topic.

kafka-topics.bat --create --bootstrap-server localhost:9092 --topic 
<TOPIC_NAME> --partitions 1 --replication-factor 1

• Kafka Topics should always have a unique name for differentiating and uniquely identifying other topics to be created in the future. In this case, you are giving a “Topic Test” as a unique name to the Topic. This name can be used to configure or customize the topic elements in further steps.
• You can list the previously created Kafka Topics using the command given below.

kafka-topics.bat  --bootstrap-server localhost:9092 -list

• Run the command to get all the partitions and replications of specific topics like a topic test.

kafka-topics.bat  --bootstrap-server localhost:9092 -describe - topic test

• Now we can see two lines in the console.

//Line#1 - We can see all the Topic’s properties like Topic Name, PartitionCount, Replication Factor, and Configs.Topic: test TopicId: fCjLNcqlSKKfC2K2m2EWIg PartitionCount: 1 Configs:
//Line#2, all partition level’s properties like partition id 0(Starting from 0), leader, and isr.Topic: test     Partition: 0 Leader: 0 Replicas: 0 Isr: 0

• After creating topics in Kafka, you can start producing and consuming messages in the further steps. By default, you have “bin/kafka-console-producer.bat” and “bin/kafka-console-consumer.bat” scripts in your main Kafka Directory.

Producer Configuration

Initially, you have to use a Kafka Producer for sending or producing Messages into the Kafka Topic. Then, you will use Kafka Consumer for receiving or consuming messages from Kafka Topics. 

Below is the configuration for the producer.

• Run the command to write data to the console for the consumer.

kafka-console-producer.bat - -broker-list localhost:9092 --topic <TOPIC_NAME>

• Add some JSON data to the topic to the console and press the enter key to send data to the topic.

{"Employee Name": "XYZ", "Employee Code": "XYZ001", "Role":" Java Developer"}

Consumer Configuration

Below is the configuration for the consumer.

• Run the following command to get data from the producer.

kafka-console-consumer.bat --topic test  --bootstrap-server localhost:9092  --from-beginning

• After sending data from the producer we will see that data which is sent by the producer as follows in the consumer cmd console.

{"Employee Name": "XYZ", "Employee Code": "XYZ001", "Role":" Java Developer"}

Conclusion

In this blog, you have learned about Apache Kafka and Apache Kafka Topics and how to configure them. You have learned the Manual Method of creating Topics and customizing Topic Configurations in Apache Kafka by the command-line tool or command prompt.

However, you can also use the Kafka Admin API, i.e., TopicBuilder Class, to programmatically implement the Topic Creation operations and it’s configuration.