AnsweredAssumed Answered

Kafka Connect Issue (NPE)

Question asked by mandoskippy on Aug 10, 2017
Latest reply on Aug 21, 2017 by madumoulin

Hey all, I am trying to do take json records into Parquet in Kafka Connect MapR Streams... 

 

I am taking baby steps (I know you can see other threads where I walk through some issues) this one is a head scratcher, I post my connect information, and when I do, the log for the worker just throws a NPE and nothing happenings... 

Any ideas would be extremely welcome!

 

 

[2017-08-10 20:48:06,238] INFO Creating connector broconnect of type io.confluent.connect.hdfs.HdfsSinkConnector (org.apache.kafka.connect.runtime.Worker:170)

[2017-08-10 20:48:06,240] INFO Instantiated connector broconnect with version 2.0.1-mapr-1707 of type io.confluent.connect.hdfs.HdfsSinkConnector (org.apache.kafka.connect.runtime.Worker:183)

[2017-08-10 20:48:06,248] INFO HdfsSinkConnectorConfig values:

filename.offset.zero.pad.width = 10

topics.dir = /data/prod/weebro

flush.size = 5

timezone =

connect.hdfs.principal =

hive.home =

hive.database = default

rotate.interval.ms = 30000

retry.backoff.ms = 5000

locale =

hadoop.home =

logs.dir = /data/prod/wal

schema.cache.size = 1000

format.class = io.confluent.connect.hdfs.parquet.ParquetFormat

hive.integration = false

hdfs.namenode.principal =

hive.conf.dir =

partition.duration.ms = -1

hadoop.conf.dir =

schema.compatibility = NONE

connect.hdfs.keytab =

hdfs.url = maprfs:///

hdfs.authentication.kerberos = false

hive.metastore.uris =

partition.field.name =

kerberos.ticket.renew.period.ms = 3600000

shutdown.timeout.ms = 3000

partitioner.class = io.confluent.connect.hdfs.partitioner.DefaultPartitioner

storage.class = io.confluent.connect.hdfs.storage.HdfsStorage

path.format =

(io.confluent.connect.hdfs.HdfsSinkConnectorConfig:135)

[2017-08-10 20:48:06,249] INFO Finished creating connector broconnect (org.apache.kafka.connect.runtime.Worker:193)

[2017-08-10 20:48:06,249] INFO ConnectorConfig values:

connector.class = class io.confluent.connect.hdfs.HdfsSinkConnector

tasks.max = 1

topics = [/apps/prod/brostreams/weebro/streams/brostreams:dns]

name = broconnect

(org.apache.kafka.connect.runtime.ConnectorConfig:165)

[2017-08-10 20:48:06,477] INFO Tasks [broconnect-0, broconnet-0] configs updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder:845)

[2017-08-10 20:48:06,484] ERROR Unexpected exception in KafkaBasedLog's work thread (org.apache.kafka.connect.util.KafkaBasedLog:393)

java.lang.NullPointerException

at org.apache.kafka.connect.storage.KafkaConfigStorage.completeTaskIdSet(KafkaConfigStorage.java:558)

at org.apache.kafka.connect.storage.KafkaConfigStorage.access$1200(KafkaConfigStorage.java:143)

at org.apache.kafka.connect.storage.KafkaConfigStorage$1.onCompletion(KafkaConfigStorage.java:476)

at org.apache.kafka.connect.storage.KafkaConfigStorage$1.onCompletion(KafkaConfigStorage.java:372)

at org.apache.kafka.connect.util.KafkaBasedLog.consumeAllRecords(KafkaBasedLog.java:317)

at org.apache.kafka.connect.util.KafkaBasedLog.readToMapRStreamsLogEnd(KafkaBasedLog.java:338)

at org.apache.kafka.connect.util.KafkaBasedLog.readToLogEnd(KafkaBasedLog.java:348)

at org.apache.kafka.connect.util.KafkaBasedLog.access$300(KafkaBasedLog.java:71)

at org.apache.kafka.connect.util.KafkaBasedLog$WorkThread.run(KafkaBasedLog.java:368)

Outcomes