Kafka cluster setup

  1. Install  kafka. Both CDH and  HDP contains kafka packages.
  2. Configure kafka. Following values should be set and the file is server.properties
  • broker.id – Id of the broker i.e. an integer. Each broker in a cluster needs to have a unique id.
  • log.dirs – Directory where you want Kafka to commit its message. Not to be confused it with usual log files.
  •  port – Port on which Kafka will accept connections from producers and consumers
    zookeeper.connect – Comma separate list of ZooKeeper nodes. E.g. hostname1:port1,hostname2:port2.

3.Zokeeper should be up and running

4. Start kafka broker using CM or Ambari or with the below commands

#kafka-server-start.sh -daemon config/server.properties

5. Start a topic with name anoop  as  below

#kafka-topics –create –zookeeper zookeeperhostname:2181/kafka –replication-factor 1 –partitions 1 –topic anoop

6. Start a kafka producer

#kafka-console-producer –broker-list hostname1:9092,hostname2:9092 –topic anoop

We can use the broker hostnames with port as 9092

Once the producer is started we can try sending sample messages

7. Start the consumer as below

#kafka-console-consumer –zookeeper zookeeperhostname:2181/kafka –topic anoop

The consumer should now show the messages from the producer shell.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s