AnsweredAssumed Answered

ReportNodeFailure Error after storing data to parquet using MapR 5 Distribution

Question asked by chetkhatri on Oct 21, 2016
Latest reply on Nov 11, 2016 by aalvarez

Hello Community Member,

I am reading data 10 GB from S3 Bucket in JSON form, reading another source from parquet (2 GB), reading MongoDB Collection as a 1 GB and doing Join Transformation and storing to parquet again, after successful completion of all stages at Spark UI, spark shell shows below error stack.

 

Note: I am using 16v Core - 104 GB Master, 8vCore - 26 GB Slave1, 8vCore - 26 GB Slave2 on Google Cloud.


16/10/22 04:39:45 INFO SparkHadoopMapRedUtil: attempt_201610220436_0006_m_000346_0: Committed
16/10/22 04:39:45 INFO Executor: Finished task 346.0 in stage 6.0 (TID 4102). 1971 bytes result sent to driver
16/10/22 04:39:45 INFO TaskSetManager: Finished task 346.0 in stage 6.0 (TID 4102) in 244436 ms on localhost (1024/1024)
16/10/22 04:39:45 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool
16/10/22 04:39:45 INFO DAGScheduler: ResultStage 6 (save at <console>:41) finished in 299.484 s
16/10/22 04:39:45 INFO DAGScheduler: Job 2 finished: save at <console>:41, took 3043.750543 s
2016-10-22 04:43:43,7457 ERROR Client fs/client/fileclient/cc/client.cc:5826 Thread: 21289 rpc err No such device(19) 28.15 to 10.140.0.7:5660, fid 2070.24949.17609664, upd 1
2016-10-22 04:43:43,8655 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:735 Thread: 21289 ReportNodeFailure failed, error Connection reset by peer(104) for cid 2070, server: 10.140.0.7:5660CLDB: 10.140.0.6:7222
2016-10-22 04:43:44,9944 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:735 Thread: 21289 ReportNodeFailure failed, error Connection reset by peer(104) for cid 2070, server: 10.140.0.7:5660CLDB: 10.140.0.6:7222
2016-10-22 04:43:46,1177 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:735 Thread: 21289 ReportNodeFailure failed, error Connection reset by peer(104) for cid 2070, server: 10.140.0.7:5660CLDB: 10.140.0.6:7222
2016-10-22 04:43:46,1178 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:1611 Thread: 21289 MoveToNextCldb: No CLDB entries, cannot run, sleeping 5 seconds!

 

 

Kindly Give resolution for the same.

Outcomes