AnsweredAssumed Answered

MapR Kafka Connect HDFS save to parquet issue

Question asked by EmielKremers on Jul 12, 2018

Hi all,

We are running into an issue saving files into parquet format. We are retreiving json data from a kafka topic. Saving it into avro works. When we change the file format into parquet we get the following exception:

[2018-07-12 15:06:42,413] ERROR WorkerSinkTask{id=eos-lgi-sink-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:172)
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
        at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:517)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:288)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:198)
        at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:166)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.StackOverflowError
        at org.apache.parquet.schema.PrimitiveType.<init>(PrimitiveType.java:341)
        at org.apache.parquet.schema.PrimitiveType.<init>(PrimitiveType.java:312)
        at org.apache.parquet.avro.AvroSchemaConverter.primitive(AvroSchemaConverter.java:205)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:118)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
.
.
.
.
.
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:132)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:137)
        at org.apache.parquet.avro.AvroSchemaConverter.convertUnion(AvroSchemaConverter.java:174)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:151)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:112)
        at org.apache.parquet.avro.AvroSchemaConverter.convertField(AvroSchemaConverter.java:187)
        at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:106)

 

Any suggestions on how to resolve this?

 

Regards Emiel

Outcomes