Topic Creation Run Console Producers & Console Consumer

on

(1) Firstly start your kafka and zookeeper


(2) Go to kafka directory

 maneesh@maneesh:~$ pwd

/home/maneesh

maneesh@maneesh:~$ cd Downloads

maneesh@maneesh:~/Downloads$ ls

apache-zookeeper-3.5.7-bin  apache-zookeeper-3.5.7-bin.tar.gz  google-chrome-stable_current_amd64.deb  jdk-15.0.2_linux-x64_bin.deb  kafka_2.13-2.7.0  kafka_2.13-2.7.0.tgz

maneesh@maneesh:~/Downloads$ cd kafka_2.13-2.7.0/


(3) Create a Topic (Topic name is myTopic)

maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic myTopic partitions 1 --replication-factor 1

Created topic myTopic.

maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ 


(4) List all the topic

maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

myTopic

maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ 


(5) Check the properties of topic (How many partition and replication factor)

maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic myTopic

Topic: myTopic PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824

Topic: myTopic Partition: 0 Leader: 1 Replicas: 1 Isr: 1

maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ 

Topic: myTopic Partition: 0 Leader: 1 Replicas: 1 Isr: 1 (This line tell about partition level 

wise  properties because myTopic only one partition so only one line is present and Partition: 0 

Zero is the id of partition , id is start from zero)

Leader :1 = When we create partition of the topic then one of the broker become leader from where we perform read and write operation in that topic 

Relicas: 1 = broker 1 contains one replica of this topic

Isr:1 = This will tell how many replica in synk (means How many copy prsent same type topic

in this partition)


(6) Create a producer and send messages

maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic myTopic

>This is a message

>message 2

>message 3

>


(7) Create a consumer and receives messages in next terminal

maneesh@maneesh:~$ cd Downloads
maneesh@maneesh:~/Downloads$ cd kafka_2.13-2.7.0/
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTopic --from-beginning
This is a message
message 2
message 3

(8) List out all consumer group
maneesh@maneesh:~$ cd Downloads
maneesh@maneesh:~/Downloads$ cd kafka_2.13-2.7.0/
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
console-consumer-95957
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ 

(9) Describe the consumer
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group console-consumer-95957

GROUP                                TOPIC           PARTITION    CURRENT-OFFSET  LOG-END-OFFSET  
console-consumer-95957     myTopic         0                                       -                                  4                         
LAG                 CONSUMER-ID                                                                                                    
    -                    consumer-console-consumer-95957-1-bea79b52-8fef-4428-bef8-dd50cbf75756 
HOST               CLIENT-ID
  /127.0.0.1      consumer-console-consumer-95957-1
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ 

LOG-END-OFFSET  = Total message in group

Consumer offset-  It contains consumer related information like how many message consumer receives and from which topic message consume. It is created by kafka cluster.

(10) List out all the topic
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
__consumer_offsets
myTopic

__consumer_offsets= It created by kafka cluster



(11) Create Multiple producer
Producer -1 (which is already present)
maneesh@maneesh:~/Downloads$ cd kafka_2.13-2.7.0/
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic myTopic partitions 1 --replication-factor 1
Created topic myTopic.
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
myTopic
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic myTopic
Topic: myTopic PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: myTopic Partition: 0 Leader: 1 Replicas: 1 Isr: 1
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic myTopic
>This is a message
>message 2
>message 3
>message 4
>producer 1
>


 Producer -2 ( created  )
maneesh@maneesh:~$ cd Downloads
maneesh@maneesh:~/Downloads$ cd kafka_2.13-2.7.0/
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic myTopic
>producer 2
>


Producer -3 ( created  )
maneesh@maneesh:~$ cd Downloads
maneesh@maneesh:~/Downloads$ cd kafka_2.13-2.7.0/
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic myTopic
>producer 3
>


Consumer-
maneesh@maneesh:~$ cd Downloads
maneesh@maneesh:~/Downloads$ cd kafka_2.13-2.7.0/
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTopic --from-beginning
This is a message
message 2
message 3
message 4
producer 1
producer 2
producer 3


(12) Create Multiple Consumer
Consumer -1 (which is already present)
maneesh@maneesh:~$ cd Downloads
maneesh@maneesh:~/Downloads$ cd kafka_2.13-2.7.0/
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTopic --from-beginning
This is a message
message 2
message 3
message 4
producer 1
producer 2
producer 3
This is the new message

Consumer -2 (created new)
maneesh@maneesh:~$ cd Downloads
maneesh@maneesh:~/Downloads$ cd kafka_2.13-2.7.0/
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTopic --from-beginning
This is a message
message 2
message 3
message 4
producer 1
producer 2
producer 3
This is the new message

Consumer -2 (created new)
maneesh@maneesh:~$ cd Downloads
maneesh@maneesh:~/Downloads$ cd kafka_2.13-2.7.0/
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTopic
This is the new message

Note - When we running consumer without making consumer group then kafaka make  group by itself  and  consumer store in it.


(13) List out all consumer group
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
console-consumer-84151
console-consumer-95957
console-consumer-22501
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ 


(14)  Create  consumer by  group name
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTopic --group myConsumerGroup
message 6


(15) List out all consumer group
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTopic --from-beginning
This is a message
message 2
message 3
message 4
producer 1
producer 2
producer 3
This is the new message
any message
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
console-consumer-84151
console-consumer-22501
myConsumerGroup
maneesh@maneesh:~/Downloads/kafka_2.13-2.7.0$ 








Comments

Post a Comment