AnsweredAssumed Answered

Spark Submit using Mapr user ticket

Question asked by anshul09013 on Jan 22, 2018
Latest reply on Jan 29, 2018 by anshul09013

Hi,

In our work-flow, we create a user ticket on client machine and spark submit to MapR cluster. It has been observed that after 7 days, spark application submitted to MapR cluster crashes and logs shown in hadoop are as below:

 

2018-01-22 10:48:06,0029 ERROR Client fs/client/fileclient/cc/client.cc:6844 Thread: 6794 rpc err Connection reset by peer(104) 37.2 to 10.222.20.43:5660, fid 2206.32.131298, upd 1
2018-01-22 10:48:06,0039 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:1337 Thread: 6794 ReportNodeFailure failed, error Read-only file system(30) for cid 2206, server: 10.222.20.43:5660 CLDB: 10.222.16.117:7222
2018-01-22 10:48:06,0047 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:1337 Thread: 6794 ReportNodeFailure failed, error Read-only file system(30) for cid 2206, server: 10.222.20.43:5660 CLDB: 10.222.16.117:7222
2018-01-22 10:48:06,0050 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:3410 Thread: 6794 Received error Invalid ticket used to communicate with the server(20021) while trying to reach CLDB: 10.222.18.206:7222
2018-01-22 10:48:06,0050 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:1322 Thread: 6794 ReportNodeFailure RPC failed due to an invalid ticket to reach CLDB. Use maprlogin to obtain a ticket and retry
2018-01-22 10:48:06,0050 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:1431 Thread: 6794 ReportNodeFailure failed, error Invalid ticket used to communicate with the server(20021),Could not allocate new master for cid 2206, CLDB: 10.222.18.206:7222
2018-01-22 10:48:06,0050 ERROR Client fs/client/fileclient/cc/client.cc:6916 Thread: 6794 rpc 37.2, failing update on fid 2206.32.131298
2018-01-22 10:48:06,0050 ERROR Client fs/client/fileclient/cc/dbclient.cc:3562 Thread: 6794 TopicFeedStatProcRPC failed for table /data/mojo/pv_sanity, error = Connection reset by peer, fid 2206.32.131298
2018-01-22 10:48:06,0080 ERROR Client fs/client/fileclient/cc/client.cc:6844 Thread: 6805 rpc err Invalid ticket used to communicate with the server(20021) 35.2 to 10.222.20.43:5660, fid 2206.32.131298, upd 1

 

Seeing the logs, it looks like mapr ticket used while submitting the spark application is also used on cluster and on expiry application crashes.

 

1) For a long running job, does ticket created on client machine is also used by MapR cluster for the application summited?

2) If yes, as we can not create user ticket with expiry more than 30 days, can we submit application using service ticket?

Outcomes