- Install kafka. Both CDH and HDP contains kafka packages.
- Configure kafka. Following values should be set and the file is server.properties
- broker.id – Id of the broker i.e. an integer. Each broker in a cluster needs to have a unique id.
- log.dirs – Directory where you want Kafka to commit its message. Not to be confused it with usual log files.
- port – Port on which Kafka will accept connections from producers and consumers
zookeeper.connect – Comma separate list of ZooKeeper nodes. E.g. hostname1:port1,hostname2:port2.
3.Zokeeper should be up and running
4. Start kafka broker using CM or Ambari or with the below commands
#kafka-server-start.sh -daemon config/server.properties
5. Start a topic with name anoop as below
#kafka-topics –create –zookeeper zookeeperhostname:2181/kafka –replication-factor 1 –partitions 1 –topic anoop
6. Start a kafka producer
#kafka-console-producer –broker-list hostname1:9092,hostname2:9092 –topic anoop
We can use the broker hostnames with port as 9092
Once the producer is started we can try sending sample messages
7. Start the consumer as below
#kafka-console-consumer –zookeeper zookeeperhostname:2181/kafka –topic anoop
The consumer should now show the messages from the producer shell.