Looking for Ideas on how to use Streams in this Use Case

Discussion created by mandoskippy on May 24, 2017
Latest reply on May 31, 2017 by mandoskippy

I am looking for a nice way to utilize netflow at massive scale... obviously MapR comes to mind... but there are things that have to be figured out... that said, I found a neat repo that seems to solve many of them:


GitHub - VerizonDigital/vflow: Enterprise Network Flow Collector (IPFIX, sFlow, Netflow) 


It's designed to by multi-master, push multiple types of flow data into Kafka, with a pluggable MQ interface (currently it supports NSQ and Kafka via the Spotify Go Package Sarama.  There are multiple questions related to this topic on the forums... but as of now, I am a bit stuck... Sarama doesn't have a MapR equivalent, because I believe it only supports the Kafka Wire protocol. Confluent has a librdkafka wrapper for Go (that I've used for Python scripts) however, it requires librdkafka 0.9.6 (which MapR is way behind with 0.9.1). 


I dont see any direct bindings for Go (and I am a very poor Go programmer, just starting). 


So I am looking for ideas on how to write an MQ plugin for vflow that would allow me to write directly to streams using GoLang. Any assistance would be great!