AnsweredAssumed Answered

Error when inserting data to DB

Question asked by NOC on Jun 29, 2017
Latest reply on Aug 8, 2017 by NOC

Hello

I'm writing a simple spark app that takes JSON files and insert them to the DB to later be queried by Drill

 

The code is rather simple (python)

 

from pyspark.sql import SparkSession

spark_job = SparkSession.builder.appName("Test").getOrCreate()
df = spark.read.json('test.json')
df.coalesce(2)
df.write.mode('append').partitionBy('date').parquet(hdfs_file_path)
spark_job.stop()

 

This works and I can see the data with drill

But I get the following errors when inserting the data (line 6)

 

17/06/29 14:56:29 ERROR MapRFileSystem: Failed to delete path hdfs://user/mapr/data/table/_temporary-d623aec0-ba23-4a04-9403-96750ea2d358, error: No such file or directory (2)

 

Any idea why?

Outcomes