AnsweredAssumed Answered

Occasional getting  "Could not create FileClient , Connection reset by peer(104) "

Question asked by oae on Mar 26, 2015
Latest reply on Apr 1, 2016 by maprcommunity
Branched to a new discussion
Having mapr installed on a 2-node ec2 system.
I'm getting occasional exceptions in my java client program:

    2015-03-26 08:35:14,5154 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:1586 Thread: 23016 MoveToNextCldb: No CLDB entries, cannot run, sleeping 5 seconds!
    2015-03-26 08:35:19,5156 ERROR Client fs/client/fileclient/cc/client.cc:813 Thread: 23016 Failed to initialize client for cluster maprCluster, error Connection reset by peer(104)
     Caused by: java.io.IOException: Could not create FileClient
         at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:527)
         at com.mapr.fs.MapRFileSystem.lookupClient(MapRFileSystem.java:588)
         at com.mapr.fs.MapRFileSystem.makeDir(MapRFileSystem.java:1066)
         at com.mapr.fs.MapRFileSystem.mkdirs(MapRFileSystem.java:1097)
         at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1851)
       
Same problem when i access the file system via command line. Here a successful attempt:

    [root@ip-10-144-184-47 ~]# hadoop fs -Dfs.mapr.trace=debug -ls /
    Setting continuous mode
    2015-03-26 09:47:01,2870 Program: fileclient on Host: NULL IP: 0.0.0.0, Port: 0, PID: 0
    2015-03-26 09:47:01,3030 DEBUG Client fs/client/fileclient/cc/client.cc:4713 Thread: 28741 User buffersize = 1024
    2015-03-26 09:47:01,3030 DEBUG Client fs/client/fileclient/cc/client.cc:4721 Thread: 28741 Group buffersize = 1024
    2015-03-26 09:47:01,3034 DEBUG Client fs/client/fileclient/cc/client.cc:4752 Thread: 28741 PutBuffer memory threshold = 33554432 MB, flush interval = 3 secs, bufferSize =  131072 bytes
    2015-03-26 09:47:01,3034 DEBUG Client fs/client/fileclient/cc/client.cc:4763 Thread: 28741 BulkLoader queueSz= 0MB flags=0
    2015-03-26 09:47:01,3035 DEBUG JniCommon fs/client/fileclient/cc/jni_common.cc:355 Thread: 28741 GetUserName: Inserting to uid-table root, 0
    2015-03-26 09:47:01,3305 DEBUG JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:515 Thread: 28741  -- Enter JNI OpenClient --
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:307 Thread: 28741 InitCreds: number of groups = 7
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:345 Thread: 28741 InitCreds: default user ID = 0
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28741 Added gid 0
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:353 Thread: 28741 effective gid 0 found in the group list
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28741 Added gid 1
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28741 Added gid 2
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28741 Added gid 3
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28741 Added gid 4
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28741 Added gid 6
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28741 Added gid 10
    2015-03-26 09:47:01,3307 DEBUG Client fs/client/fileclient/cc/client.cc:492 Thread: 28741 Security not enabled
    2015-03-26 09:47:01,3308 DEBUG Client fs/client/fileclient/cc/client.cc:5289 Thread: 28741 CheckImpersonation: Impersonation is disabled; all attempts at impersonation will use the process user
    2015-03-26 09:47:01,3308 DEBUG Client fs/client/fileclient/cc/client.cc:561 Thread: 28741 Init: cluster maprCluster pid 28715, CLDB 10.231.41.89:7222, numCLDBs 1
    2015-03-26 09:47:01,3311 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:1242 Thread: 28741 Requesting Volume mapr.cluster.root srcCluster null vtype 2 wantMirror 1
    2015-03-26 09:47:01,3311 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:1250 Thread: 28741 Requesting for cldb information
    2015-03-26 09:47:01,3332 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:1346 Thread: 28741 Received Cldb Ips from 10.231.41.89:7222
    2015-03-26 09:47:01,3332 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:1359 Thread: 28741 Received Cldb Ip : 10.231.41.89:7222
    2015-03-26 09:47:01,3332 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:1406 Thread: 28741 Setting myTopology to /data/default-rack/ip-10-144-184-47
    2015-03-26 09:47:01,3332 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:349 Thread: 28741 Created new entry for cid 2049
    2015-03-26 09:47:01,3333 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:115 Thread: 28741 PopulateEntry: For CID 2049 received host IP 10.144.184.47
    2015-03-26 09:47:01,3333 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:158 Thread: 28741 PopulateEntry: Mismatch of server index 0 of binding 0x7fc8bcaec930, for peer 10.144.184.47:5660
    2015-03-26 09:47:01,3333 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:186 Thread: 28741 PopulateEntry: Adding binding 0x7fc8bcaec930 with server index 1 for peer with 1 IPs:
    2015-03-26 09:47:01,3333 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:191 Thread: 28741 PopulateEntry: binding <0x7fc8bcaec930> IP address: 10.144.184.47:5660
    2015-03-26 09:47:01,3334 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:115 Thread: 28741 PopulateEntry: For CID 2049 received host IP 10.231.41.89
    2015-03-26 09:47:01,3334 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:158 Thread: 28741 PopulateEntry: Mismatch of server index 0 of binding 0x7fc8bcaecbe0, for peer 10.231.41.89:5660
    2015-03-26 09:47:01,3334 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:186 Thread: 28741 PopulateEntry: Adding binding 0x7fc8bcaecbe0 with server index 2 for peer with 1 IPs:
    2015-03-26 09:47:01,3334 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:191 Thread: 28741 PopulateEntry: binding <0x7fc8bcaecbe0> IP address: 10.231.41.89:5660
    2015-03-26 09:47:01,3334 INFO Client fs/client/fileclient/cc/client.cc:586 Thread: 28741 Populating Server Ticket And Key
    2015-03-26 09:47:01,3334 INFO Client fs/client/fileclient/cc/client.cc:593 Thread: 28741 Populated the server key and ticket successfully
    2015-03-26 09:47:01,3334 DEBUG Client fs/client/fileclient/cc/client.cc:622 Thread: 28741 Init: cluster maprCluster, CLDB 10.231.41.89:7222

And one unsuccessful one:

    [root@ip-10-144-184-47 ~]# hadoop fs -Dfs.mapr.trace=debug -ls /
    Setting continuous mode
    2015-03-26 09:45:00,3811 Program: fileclient on Host: NULL IP: 0.0.0.0, Port: 0, PID: 0
    2015-03-26 09:45:00,3970 DEBUG Client fs/client/fileclient/cc/client.cc:4713 Thread: 28189 User buffersize = 1024
    2015-03-26 09:45:00,3970 DEBUG Client fs/client/fileclient/cc/client.cc:4721 Thread: 28189 Group buffersize = 1024
    2015-03-26 09:45:00,3974 DEBUG Client fs/client/fileclient/cc/client.cc:4752 Thread: 28189 PutBuffer memory threshold = 33554432 MB, flush interval = 3 secs, bufferSize =  131072 bytes
    2015-03-26 09:45:00,3974 DEBUG Client fs/client/fileclient/cc/client.cc:4763 Thread: 28189 BulkLoader queueSz= 0MB flags=0
    2015-03-26 09:45:00,3975 DEBUG JniCommon fs/client/fileclient/cc/jni_common.cc:355 Thread: 28189 GetUserName: Inserting to uid-table root, 0
    2015-03-26 09:45:00,4243 DEBUG JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:515 Thread: 28189  -- Enter JNI OpenClient --
    2015-03-26 09:45:00,4244 DEBUG Client fs/client/fileclient/cc/client.cc:307 Thread: 28189 InitCreds: number of groups = 7
    2015-03-26 09:45:00,4245 DEBUG Client fs/client/fileclient/cc/client.cc:345 Thread: 28189 InitCreds: default user ID = 0
    2015-03-26 09:45:00,4245 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28189 Added gid 0
    2015-03-26 09:45:00,4245 DEBUG Client fs/client/fileclient/cc/client.cc:353 Thread: 28189 effective gid 0 found in the group list
    2015-03-26 09:45:00,4245 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28189 Added gid 1
    2015-03-26 09:45:00,4245 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28189 Added gid 2
    2015-03-26 09:45:00,4245 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28189 Added gid 3
    2015-03-26 09:45:00,4245 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28189 Added gid 4
    2015-03-26 09:45:00,4245 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28189 Added gid 6
    2015-03-26 09:45:00,4245 DEBUG Client fs/client/fileclient/cc/client.cc:351 Thread: 28189 Added gid 10
    2015-03-26 09:45:00,4245 DEBUG Client fs/client/fileclient/cc/client.cc:492 Thread: 28189 Security not enabled
    2015-03-26 09:45:00,4246 DEBUG Client fs/client/fileclient/cc/client.cc:5289 Thread: 28189 CheckImpersonation: Impersonation is disabled; all attempts at impersonation will use the process user
    2015-03-26 09:45:00,4246 DEBUG Client fs/client/fileclient/cc/client.cc:561 Thread: 28189 Init: cluster maprCluster pid 28163, CLDB 10.231.41.89:7222, numCLDBs 1
    2015-03-26 09:45:00,4250 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:1242 Thread: 28189 Requesting Volume mapr.cluster.root srcCluster null vtype 2 wantMirror 1
    2015-03-26 09:45:00,4250 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:1250 Thread: 28189 Requesting for cldb information
    2015-03-26 09:45:00,4256 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:913 Thread: 28189 Lookup of volume mapr.cluster.root failed, error Connection reset by peer(104), CLDB: 10.231.41.89:7222
    2015-03-26 09:45:00,4256 DEBUG Client fs/client/fileclient/cc/client.cc:607 Thread: 28189 Init: Failed to initialize cidCache for cluster maprCluster, CLDB 10.231.41.89:7222, error 104
    2015-03-26 09:45:00,4256 ERROR Client fs/client/fileclient/cc/client.cc:394 Thread: 28189 Failed to initialize client for cluster maprCluster, error Connection reset by peer(104)
    ls: Could not create FileClient
    2015-03-26 09:45:00,4257 DEBUG Client fs/client/fileclient/cc/client.cc:519 Thread: 28189 MapClient object destructor
    2015-03-26 09:45:00,4257 DEBUG Client fs/client/fileclient/cc/client.cc:541 Thread: 28189 MapClient object destroyed
    2015-03-26 09:45:00,4257 DEBUG JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:535 Thread: 28189  -- Exit JNI OpenClient --

 
Any idea whats wrong ?

That is with mapr-4.0.1 in yarn mode.

Johannes

Outcomes