How to install and configure Kafka on MapR

Document created by wade on Feb 27, 2016
Version 1Show Document
  • View in full screen mode

Author: Jitendra Yadav, last modified by Bo soon Park on October 26, 2015

Original Publication Date: June 17, 2015

 

Environment

MapR 3.1.1

Goal

How to install and configure Kafka on MapR.

Solution

1. Download and install scala-2.10.4

http://www.scala-lang.org/files/archive/scala-2.10.4.rpm

rpm -ivh scala-2.10.4.rpm

2. Download and extract Kafka

http://mirror.cogentco.com/pub/apache/kafka/0.8.1.1/kafka_2.10-0.8.1.1.tgz

tar -xzf kafka_2.10-0.8.1.1.tgz

3. Move the extracted directory to /opt/kafka.

4. Modify /opt/kafka/config/server.properties to set log.dirs and also reflect zookeeper quorum settings.

log.dirs=/opt/kafka/logs

Zookeeper connection string can be found using “maprcli node list zookeepers”, then set :

zookeeper.connect=maprtestvm:5181

5. Start Kafka server

cd /opt/kafka/ 
bin/kafka-server-start.sh config/server.properties > logs/kafka-server.log 2>&1 &

6. Create a topic called “hbasesets”

bin/kafka-topics.sh --zookeeper maprtestvm:5181 --create --topic hbasetest --partitions 3 --replication-factor 1

7. Verify the topic has been created

bin/kafka-topics.sh --zookeeper maprtestvm:5181 --list

8. List Kafka topic details and partitions

bin/kafka-topics.sh --zookeeper maprtestvm:5181 --describe --topic hbasetest  

Topic:hbasetest PartitionCount:3 ReplicationFactor:1 Configs:
Topic: hbasetest        Partition: 0    Leader: 0       Replicas: 0     Isr: 0
Topic: hbasetest        Partition: 1    Leader: 0       Replicas: 0     Isr: 0
Topic: hbasetest        Partition: 2    Leader: 0       Replicas: 0     Isr: 0

9. Start a producer

bin/kafka-console-producer.sh --broker-list maprtestvm:9092 --sync --topic hbasetest 2>&1 | tee logs/kafka-producer1.log

 

 

Attachments

    Outcomes