AnsweredAssumed Answered

Error while reading data from multiple MapR topics simultaneously

Question asked by anshul09013 on Aug 23, 2017
Latest reply on Sep 28, 2017 by nagrawal

Hi,

I have a spark job which is reading data from 4000 topics. I have a 5 node cluster with 8 cores each. Application fails while reading data with an error below:

 

17/08/23 14:43:53 WARN ExecutorAllocationManager: Unable to reach the cluster manager to kill executor/s executorIdsToBeRemoved.mkString(",") or no executor eligible to kill!
17/08/23 14:43:54 WARN ExecutorAllocationManager: Unable to reach the cluster manager to kill executor/s executorIdsToBeRemoved.mkString(",") or no executor eligible to kill!
17/08/23 14:43:54 WARN ExecutorAllocationManager: Unable to reach the cluster manager to kill executor/s executorIdsToBeRemoved.mkString(",") or no executor eligible to kill!
17/08/23 14:43:55 WARN ExecutorAllocationManager: Unable to reach the cluster manager to kill executor/s executorIdsToBeRemoved.mkString(",") or no executor eligible to kill!
2017-08-23 14:43:57,6548 ERROR Client fs/client/fileclient/cc/client.cc:6503 Thread: 28266 rpc err Connection reset by peer(104) 37.4 to 10.222.18.12:5660, fid 2128.216.132498, upd 1
2017-08-23 14:43:57,6549 ERROR Client fs/client/fileclient/cc/client.cc:6503 Thread: 28270 rpc err Connection reset by peer(104) 37.4 to 10.222.18.12:5660, fid 2128.200.132466, upd 1
2017-08-23 14:43:57,6549 ERROR Client fs/client/fileclient/cc/client.cc:6503 Thread: 28269 rpc err Connection reset by peer(104) 37.4 to 10.222.18.12:5660, fid 2318.65.131424, upd 1
2017-08-23 14:43:57,6550 ERROR Client fs/client/fileclient/cc/client.cc:6503 Thread: 28272 rpc err Connection reset by peer(104) 37.4 to 10.222.18.12:5660, fid 2128.193.132452, upd 1
2017-08-23 14:43:57,6550 ERROR Client fs/client/fileclient/cc/client.cc:6503 Thread: 28265 rpc err Connection reset by peer(104) 37.4 to 10.222.18.12:5660, fid 2140.192.131616, upd 1
2017-08-23 14:43:57,6551 ERROR Client fs/client/fileclient/cc/client.cc:6503 Thread: 29075 rpc err Connection reset by peer(104) 35.2 to 10.222.18.12:5660, fid 2278.105.131568, upd 1
2017-08-23 14:43:57,6551 ERROR Client fs/client/fileclient/cc/client.cc:6503 Thread: 28267 rpc err Connection reset by peer(104) 37.4 to 10.222.18.12:5692, fid 2241.62.131412, upd 1
2017-08-23 14:43:57,6552 ERROR Client fs/client/fileclient/cc/client.cc:6503 Thread: 28487 rpc err Connection reset by peer(104) 37.2 to 10.222.18.12:5660, fid 2318.72.131438, upd 1
2017-08-23 14:43:57,6553 ERROR Client fs/client/fileclient/cc/client.cc:6503 Thread: 28268 rpc err Connection reset by peer(104) 37.4 to 10.222.18.12:5660, fid 2116.91.131420, upd 1
2017-08-23 14:43:57,6553 ERROR Client fs/client/fileclient/cc/client.cc:6503 Thread: 28271

 

Detailed logs are attached.

 

Is it because we are reading from multiple topics simultaneously?

Attachments

Outcomes