AnsweredAssumed Answered

Keeping up with librdkafka

Question asked by mandoskippy on May 8, 2017
Latest reply on May 31, 2018 by maprcommunity

I went to go try playing with with librdkafka to use streams today.  My goal was to get something that used librdkafka working with the MapR librdkafka. While I am not good at any of this stuff, I did once stay at a Holiday Inn Express. 

 

In summary, I found that I MapR likely used version 0.9.1 as a starting base for their librdkafka.  While Kafkacat (currently) uses 0.9.5.   In my git repo:

 

https://github.com/JohnOmernik/maprkafkacat

 

I have some build scripts (all in docker so easy to replicate what I am doing).  If you run

 

./build_std.sh

 

It will use librdkafka version 0.9.5 and build both librdkafka and kafkacat correctly. 

 

If you try to use 

 

./build_mapr.sh

 

It will produce a weird error, and when I made an issue on the kafkacat git, the response was "it looks like you are using an older version of librdkafka, can you upgrade?"

 

To validate, I created another script

 

./build_std_old.sh

 

Where I use the standard librdkafka version 0.9.1 (Which I suspect is what MapR is built off of ) and sure enough, it fails with the same error as the MapR version. 

 

So, my question is this, will mapr look to keep in sync with librdkafka? For it to be useful, and truly a portable API, I think we need some understanding of how in sync/ out of sync MapR's librdkafka will be... 

 

I am looking forward to discussions on this topic!

 

John

Outcomes