AnsweredAssumed Answered

Issues with MapR Kafka Connect for Streams

Question asked by mandoskippy on Aug 9, 2017
Latest reply on Aug 10, 2017 by cathy

Hello all, I am trying to dig through the documentation on MapR's Kafka Connect implementation. Granted, I am trying to run this in a Docker Container, however, at this point it should still be working.  I get to the point where I start a worker in distributed mode (I am only trying one worker as of now, I will deal with connectivity later).  

 

The start-distributed works, but then just exits. I go to the log, and you can see that below. I am not sure how to interpret this, but I believe there is some sort of version mismatch. 

 

Somethings to note:

 

1. I have created the topics for distributed mode, and I believe they are being connected to correctly. 

2. I am running MapR  5.2.1.42646 GA, but I am doing some custom stuff there. (I am mapping in the conf folder, the lib folder, the hadoop folder, bin folder and the include folder into my docker container, while the mapr kafka and kafka connect files are the exact files from the .deb. I do create links from /usr/bin to the MapR stuff as needed.  The reasoning here is every node will have the core MapR libs as a client, but the kafka stuff I want to live in the container for versioning etc)

3. That said, 5.2.1 seems to be a MapR version requirement (not Kafka even though the error messages reference Kafka) Where is it getting that version? How do I give it the version info it needs (If it's just a case of not getting the proper MapR version through some check)

 

4. ?????

5. Profit. 

 

Any help would be appreciated!

 

 

 

 

 

2017-08-09 18:59:55,850] INFO jetty-9.2.12.v20150709 (org.eclipse.jetty.server.Server:327)

[2017-08-09 18:59:55,861] INFO Finished reading KafakBasedLog for topic /apps/prod/broconnect/streams/__mapr_connect:configs (org.apache.kafka.connect.util.KafkaBasedLog:150)

[2017-08-09 18:59:55,861] INFO Started KafakBasedLog for topic /apps/prod/broconnect/streams/__mapr_connect:configs (org.apache.kafka.connect.util.KafkaBasedLog:152)

[2017-08-09 18:59:55,861] INFO Started KafkaConfigStorage (org.apache.kafka.connect.storage.KafkaConfigStorage:242)

[2017-08-09 18:59:55,861] INFO Herder started (org.apache.kafka.connect.runtime.distributed.DistributedHerder:156)

[2017-08-09 18:59:55,879] ERROR Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:166)

org.apache.kafka.common.errors.BrokerNotAvailableException: Feature not available on server. Please upgrade to at least Version 5.2.1

[2017-08-09 18:59:56,212] INFO Started o.e.j.s.ServletContextHandler@a5b0b86{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744)

[2017-08-09 18:59:56,220] INFO Started ServerConnector@7fc44dec{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:266)

[2017-08-09 18:59:56,220] INFO Started @1817ms (org.eclipse.jetty.server.Server:379)

[2017-08-09 18:59:56,220] INFO REST server listening at http://172.17.0.4:8083/, advertising URL http://172.17.0.4:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:132)

[2017-08-09 18:59:56,220] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:60)

[2017-08-09 18:59:56,221] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:68)

[2017-08-09 18:59:56,223] INFO Stopped ServerConnector@7fc44dec{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:306)

[2017-08-09 18:59:56,227] INFO Stopped o.e.j.s.ServletContextHandler@a5b0b86{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:865)

[2017-08-09 18:59:56,228] INFO Herder stopping (org.apache.kafka.connect.runtime.distributed.DistributedHerder:310)

[2017-08-09 18:59:56,228] INFO Herder stopped (org.apache.kafka.connect.runtime.distributed.DistributedHerder:331)

[2017-08-09 18:59:56,228] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:115)

[2017-08-09 18:59:56,228] INFO Stopping KafkaOffsetBackingStore (org.apache.kafka.connect.storage.KafkaOffsetBackingStore:91)

[2017-08-09 18:59:56,228] INFO Stopping KafkaBasedLog for topic /apps/prod/broconnect/streams/__mapr_connect:offsets (org.apache.kafka.connect.util.KafkaBasedLog:156)

[2017-08-09 18:59:56,229] INFO Stopped KafkaBasedLog for topic /apps/prod/broconnect/streams/__mapr_connect:offsets (org.apache.kafka.connect.util.KafkaBasedLog:182)

[2017-08-09 18:59:56,229] INFO Stopped KafkaOffsetBackingStore (org.apache.kafka.connect.storage.KafkaOffsetBackingStore:93)

[2017-08-09 18:59:56,229] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:155)

[2017-08-09 18:59:56,229] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:74)

Outcomes