I went to go try playing with with librdkafka to use streams today. My goal was to get something that used librdkafka working with the MapR librdkafka. While I am not good at any of this stuff, I did once stay at a Holiday Inn Express.
In summary, I found that I MapR likely used version 0.9.1 as a starting base for their librdkafka. While Kafkacat (currently) uses 0.9.5. In my git repo:
I have some build scripts (all in docker so easy to replicate what I am doing). If you run
It will use librdkafka version 0.9.5 and build both librdkafka and kafkacat correctly.
If you try to use
It will produce a weird error, and when I made an issue on the kafkacat git, the response was "it looks like you are using an older version of librdkafka, can you upgrade?"
To validate, I created another script
Where I use the standard librdkafka version 0.9.1 (Which I suspect is what MapR is built off of ) and sure enough, it fails with the same error as the MapR version.
So, my question is this, will mapr look to keep in sync with librdkafka? For it to be useful, and truly a portable API, I think we need some understanding of how in sync/ out of sync MapR's librdkafka will be...
I am looking forward to discussions on this topic!