AnsweredAssumed Answered

Pig fail in mapreduce mode

Question asked by waltari2001 on Jul 12, 2012
Hi,

I can run my pig script in local mode just fine but will fail in mapreduce mode with error:

    java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.SnappyCodec not found.
     at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:116)
     at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:156)
     at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:70)
     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:176)
     at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:458)
     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:653)
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:396)
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
     at org.apache.hadoop.mapred.Child.main(Child.java:264)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.compress.SnappyCodec
     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
     at java.security.AccessController.doPrivileged(Native Method)
     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
     at java.lang.Class.forName0(Native Method)
     at java.lang.Class.forName(Class.java:247)
     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1028)
     at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:109)
     ... 11 more

Looking at the 'failed' job params I can see:

    io.compression.codecs org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec


org.apache.hadoop.io.compress.SnappyCodec was never there before. And it's not even listed in .jtconf:

    /opt/mapr/hadoop/hadoop-0.20.2/.jtconf:
    <property><!--Loaded from core-default.xml--><name>io.compression.codecs</name><value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec</value></property>

Thanks,

Sadek


Outcomes