AnsweredAssumed Answered

SnappyCodec not found - MapR Client for Windows

Question asked by mlauer on Sep 11, 2012
Latest reply on Sep 13, 2012 by gera
Hello,

I am attempting to run a Map Reduce job (WordCount) using the MapR Client for Windows.  I am able to run this job (same command line paramaters and all) on the MapR cluster itself, and I am able to use the MapR Client to browse the file system, upload files, add/delete directories, etc. 

When I attempt to run Map Reduce jobs from the Windows client command prompt, however, using

    C:\opt\mapr\hadoop\hadoop-0.20.2\bin>hadoop jar WordCount.jar WordCount /stage/wordcount /stage/wordcount/output

I get the following error from several cluster nodes.

     12/09/11 17:05:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
     12/09/11 17:05:12 WARN snappy.LoadSnappy: Snappy native library not loaded
     12/09/11 17:05:12 INFO mapred.JobClient: Creating job's output directory at /stage/wordcount/output
     12/09/11 17:05:12 INFO mapred.JobClient: Creating job's user history location directory at /stage/wordcount/output/_logs
     12/09/11 17:05:13 INFO mapred.JobClient: Running job: job_201209071143_0078
     12/09/11 17:05:14 INFO mapred.JobClient:  map 0% reduce 0%
     12/09/11 17:05:22 INFO mapred.JobClient: Task Id : attempt_201209071143_0078_m_000000_0, Status : FAILED on node mapr1
     java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.SnappyCodec not found.
             at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:116)
             at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:156)
             at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:70)
             at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:469)
             at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:667)
             at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
             at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
             at java.security.AccessController.doPrivileged(Native Method)
             at javax.security.auth.Subject.doAs(Subject.java:396)
             at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
             at org.apache.hadoop.mapred.Child.main(Child.java:264)
     Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.compress.SnappyCodec
             at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
             at java.security.AccessController.doPrivileged(Native Method)
             at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
             at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
             at java.lang.Class.forName0(Native Method)
             at java.lang.Class.forName(Class.java:247)
             at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1028)
             at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:109)*     12/09/11 17:05:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
     12/09/11 17:05:12 WARN snappy.LoadSnappy: Snappy native library not loaded
     12/09/11 17:05:12 INFO mapred.JobClient: Creating job's output directory at /stage/wordcount/output
     12/09/11 17:05:12 INFO mapred.JobClient: Creating job's user history location directory at /stage/wordcount/output/_logs
     12/09/11 17:05:13 INFO mapred.JobClient: Running job: job_201209071143_0078
     12/09/11 17:05:14 INFO mapred.JobClient:  map 0% reduce 0%
     12/09/11 17:05:22 INFO mapred.JobClient: Task Id : attempt_201209071143_0078_m_000000_0, Status : FAILED on node mapr1
     java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.SnappyCodec not found.
             at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:116)
             at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:156)
             at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:70)
             at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:469)
             at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:667)
             at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
             at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
             at java.security.AccessController.doPrivileged(Native Method)
             at javax.security.auth.Subject.doAs(Subject.java:396)
             at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
             at org.apache.hadoop.mapred.Child.main(Child.java:264)
     Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.compress.SnappyCodec
             at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
             at java.security.AccessController.doPrivileged(Native Method)
             at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
             at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
             at java.lang.Class.forName0(Native Method)
             at java.lang.Class.forName(Class.java:247)
             at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1028)
             at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:109)

I am wondering if I need to add additional libraries to the cluster or the Windows client, reference specific classpaths when I run the Map Reduce from the Windows client, add codec references to the site xml files, etc.  Really not sure why this runs on the cluster fine, but not from the Windows client when all other MapR file system browsing capabilities seems to work just fine.

Any help would be greatly appreciated.  Thanks!

Matt

Outcomes