Troubleshooting RPC Connection reset by peer(104) 28.122 to 0.0.0.0:5660

Document created by wade on Feb 27, 2016
Version 1Show Document
  • View in full screen mode

Author: Jitendra Yadav, last modified by Sanjamala Nayeem on May 7, 2015

Original Publication Date: May 7, 2015

 

 

Environment

 

MapR 2.x, 3.x ,4.x

 

Symptom

 

While doing any read/write operation on MapRFS, the clent needs to connect to all the MFS nodes where the desired name or data containers exist. This problem comes when if client is not able to perform read/write ops on a MFS node, In that case we might see below errors at client side.

 

139721763559168 rpc err Connection reset by peer(104) 28.122 to 10.10.10.11:5660, fid 13156.251.593598, upd 0 
2015-04-20 20:37:35,3602 ERROR Client fs/client/fileclient/cc/client.cc:3468 Thread: 139721763559168 rpc err Connection reset by peer(104) 28.122 to 10.10.10.11:5660, fid 7344.1198.432180, upd 0
2015-04-20 20:38:56,5342 ERROR Client fs/client/fileclient/cc/client.cc:3468

 

Root Cause

The error "Connection reset by peer(104) 28.122 to 10.10.10.11:5660,, fid 7344.1198.432180, upd 0"  means errno 104 ECONNRESET which is system call on a socket. So it means that the node 10.10.10.11:5660 is dead or not reachable.

 

The "upd 0" means this is an attempt to fetch data from the server (as opposed to writing data to it).

 

Solution

 

1. Need to check whether the node 10.10.10.11 is reachable or not, if not then troubleshoot OS/network part.

 

ping 10.10.10.11

 

2. Check if mfs is running on node 10.10.10.11 or not. If not then restart the MFS service.

 

ps -ef | grep mfs

telnet 10.10.10.11:5660

 

3. It might also worth to check if mfs got crashed last time. If recent core available under /opt/cores with prefix "mfs"  then there was a mfs crash happened. In that situation please collect support dump, core trace and core file and open support ticket.

 

ls -l /opt/cores.

Attachments

    Outcomes