AnsweredAssumed Answered

MapR Kafka Connect HDFS issue

Question asked by Géry on May 2, 2018
Latest reply on May 3, 2018 by maprcommunity

Hi,

 

I'm trying to push data from kafka to mapr-FS.

I'm using mapr 5.2.0 with MapR kafka bundle 0.9.0 and MapR kafka-connect-hdfs 2.0.1

Kafka server and connect-standalone are running on one node of our MapR cluster.

 

I tried information provided inKafka Connect HDFS sink issue  with no success

 

I'm using file source, file sink and hdfs sink connectors.

 

Here is my configuration

connect-standalone.properties

bootstrap.servers=localhost:9092

key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000

connect-file-source.properties

name=local-file-source
connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
tasks.max=1
file=test.txt
topic=connect-test

connect-file-sink.properties

name=local-file-sink
connector.class=org.apache.kafka.connect.file.FileStreamSinkConnector
tasks.max=1
file=test.sink.txt
topics=connect-test

quickstart-hdfs.properties

name=hdfs-sink
connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
tasks.max=1
topics=connect-test
hdfs.url=maprfs:///
flush.size=1

 

I'm launching connect with

/opt/mapr/kafka/kafka-0.9.0/bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.prorties config/quickstart-hdfs.properties

 

I'm also running a console-consumer

 /opt/mapr/kafka/kafka-0.9.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning --zookeeper zookeeper_addr

 

Pushing data in kafka throught test.txt file is working and is consumed by console-consumer

echo hello >> test.txt

produces in console-consumer

{"schema":{"type":"string","optional":false},"payload":"hello"}

 

but

  • test.sink.txt is not created
  • no directory is created in maprfs. I was expecting /topics/connect-test

 

Connect logfile is attached.

 

Any help is appreciated

 

Thanks

Outcomes