AnsweredAssumed Answered

MapR Streams doesn't work with fluentd

Question asked by mkiuchi on Mar 7, 2016
Latest reply on Apr 4, 2016 by mufeed

Hi,

I've tried to publish fluentd to MapR Streams, but it fails.

Any advice? Here is log of my trial.

 

Fluentd error log

2016-03-08 16:31:20 +0900 [info]: brokers has been refreshed via Zookeeper: ["maprcd.mkhome:9092"] 2016-03-08 16:31:20 +0900 [info]: initialized producer kafka 2016-03-08 16:31:20 +0900 [info]: following tail of /usr/tmp/test.pcap 2016-03-08 16:31:40 +0900 [warn]: Send exception occurred: Poseidon::Errors::UnableToFetchMetadata 2016-03-08 16:31:40 +0900 [info]: brokers has been refreshed via Zookeeper: ["maprcd.mkhome:9092"] 2016-03-08 16:31:40 +0900 [info]: initialized producer kafka 2016-03-08 16:31:40 +0900 [warn]: emit transaction failed: error_class=Poseidon::Errors::UnableToFetchMetadata error="Poseidon::Errors::UnableToFetchMetadata" tag="test.pcap"   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/poseidon-0.0.5/lib/poseidon/sync_producer.rb:94:in `ensure_metadata_available_for_topics'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/poseidon-0.0.5/lib/poseidon/sync_producer.rb:48:in `send_messages'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/poseidon-0.0.5/lib/poseidon/producer.rb:163:in `send_messages'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-kafka-0.1.3/lib/fluent/plugin/out_kafka.rb:153:in `block in emit'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event.rb:128:in `call'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event.rb:128:in `block in each'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event.rb:127:in `each'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event.rb:127:in `each'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-kafka-0.1.3/lib/fluent/plugin/out_kafka.rb:145:in `emit'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/event_router.rb:88:in `emit_stream'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_tail.rb:248:in `receive_lines'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_tail.rb:343:in `call'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_tail.rb:343:in `wrap_receive_lines'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_tail.rb:536:in `call'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_tail.rb:536:in `on_notify'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_tail.rb:369:in `on_notify'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_tail.rb:470:in `call'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_tail.rb:470:in `on_change'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/loop.rb:88:in `run_once'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/loop.rb:88:in `run'   2016-03-08 16:31:40 +0900 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_tail.rb:233:in `run'

MapR side

maprcli stream create -path /mkstreamtest1 maprcli stream topic create -path /mkstreamtest1 -topic topic1

 

edit /opt/mapr/kafka/kafka-0.9.0/config/server.properties to see localhost:5181 as zookeeper

run /opt/mapr/kafka/kafka-0.9.0/bin/kafka-server-start.sh /opt/mapr/kafka/kafka-0.9.0/config/server.properties

 

test connection works well

/opt/mapr/kafka/kafka-0.9.0/bin/kafka-console-consumer.sh --topic /mkstreamtest1:topic1 --new consumer --bootstrap-server localhost:9092 /opt/mapr/kafka/kafka-0.9.0/bin/kafka-console-producer.sh --topic /mkstreamtest1:topic1 --broker-list localhost:9092

When I see error on fluentd side, there is nothing output on MapR side.

 

Fluentd configuration(/etc/td-agent/td-agent.conf)

<source> type tail path /usr/tmp/test.pcap tag test.pcap format /^(?<original>.*).*$/ </source> <match *.*> @type kafka brokers maprcd.mkhome:9092 zookeeper maprcd.mkhome:5181 default_topic /mkstreamtest1:topic1 #type stdout </match>

Outcomes