AnsweredAssumed Answered

FileSystem.listFiles sometimes hangs with maprfs

Question asked by shani on Jun 27, 2018
Latest reply on Jul 1, 2018 by shani

 

Hi, I'm trying to work with MaprFS as a distributed file sytstem. I started by writing a file to the FileSystem and now I'm trying to list files, my code is written in scala (the next snippet is the first thing that I run in my main function): 

 

    val conf = new Configuration()
    val fs = FileSystem.get(conf)

    val path = new Path("maprfs:///mapr/maprdemo.mapr.io/hey/a/b/c/file.w")

    val x = fs.listFiles(path.getParent.getParent.getParent, true)
    val iterator = new Iterator[LocatedFileStatus] {
      override def hasNext: Boolean = x.hasNext

      override def next(): LocatedFileStatus = x.next()
    }.toArray

    for (file <- iterator) {
      println(file)
    }

I'm working with MaprFS 5.1.0, and I used the latest container maprtech/dev-sandbox-container:latest.

When it hangs, this is the call stack of the main thread : 

"main@1" prio=5 tid=0x1 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
       at com.mapr.fs.jni.MapRClient.getBlockInfo(MapRClient.java:-1)
       at com.mapr.fs.MapRClientImpl.getBlockLocations(MapRClientImpl.java:397)
       at com.mapr.fs.MapRFileSystem.getMapRFileBlockLocations(MapRFileSystem.java:1863)
       at com.mapr.fs.MapRFileSystem.getFileBlockLocations(MapRFileSystem.java:1883)
       at org.apache.hadoop.fs.FileSystem.getFileBlockLocations(FileSystem.java:742)
       at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1762)
       at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1747)
       at org.apache.hadoop.fs.FileSystem$6.hasNext(FileSystem.java:1830)

There are no other user threads in the process (only Finalizer, process reaper, reference handler and signal dispatcher).

 

And this error shows up after minutes: 

2018-06-27 15:10:25,1683 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:448 Thread: 21677 ContainerLookup RPC error Connection timed out(110) to CLDB 172.17.0.2:7222 for cid 2119 failed
2018-06-27 15:12:06,2549 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:448 Thread: 21677 ContainerLookup RPC error Connection timed out(110) to CLDB 10.0.0.135:7222 for cid 2119 failed

 

Also, I would like to know how to fix this error : 

18/06/27 15:08:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

This is my build.sbt : 

name := "MapRDemo"

version := "0.1"

scalaVersion := "2.12.6"

resolvers += "mapr" at "http://repository.mapr.com/maven/"

libraryDependencies += "com.mapr.hadoop" % "maprfs-core" % "5.1.0-mapr"
libraryDependencies += "com.mapr.hadoop" % "maprfs-jni" % "5.1.0-mapr"

unmanagedBase := file("/opt/mapr/lib/")

Thanks!

 

java mapr filesystem maprfs 

Outcomes