Fix - java.io.IOException: wrong value class: class org.apache.hadoop.io.Text is not class org.apache.hadoop.io.LongWritable

Document created by rsingh on Feb 13, 2016
Version 1Show Document
  • View in full screen mode

Author: Rajkumar Singh

 

Original Publication Date: April 28, 2015

 

Env:

MapR supported versions

 

Symptom:

User has a simple java Map Reduce program using combiner like this

 

public class AppDriver extends Configured implements Tool {

  

  public static class MyMapper extends Mapper<LongWritable, Text, Text, LongWritable> {

  protected void map(LongWritable key, Text value,

  Context context)

  throws IOException, InterruptedException {

  //--- some code ---

  context.write(new Text("key"), new LongWritable(1));

  }

  }

  

  public static class MyReducer extends Reducer<Text, LongWritable, Text, Text> {

  protected void reduce(Text key, Iterable<LongWritable> values, Context context)

  throws IOException, InterruptedException {

  // --- some code ---

  context.write(key, new Text("any text"));

  }

  }

  

  /**

  * @param args

  * @throws Exception

  */

  public static void main(String[] args) throws Exception {

  int exitCode = ToolRunner.run(new AppDriver(),

  args);

  System.exit(exitCode);

  @Override

  public int run(String[] args) throws Exception {

  Configuration conf = getConf();

  Job job = new Job(conf, "Combiner Job");

  job.setJarByClass(AppDriver.class);

  job.setMapperClass(MyMapper.class);

  job.setCombinerClass(MyReducer.class);

  job.setReducerClass(MyReducer.class);

  job.setMapOutputKeyClass(Text.class);

  job.setMapOutputValueClass(LongWritable.class);

  job.setOutputKeyClass(Text.class);

  job.setOutputValueClass(Text.class);

  job.setInputFormatClass(TextInputFormat.class);

  FileInputFormat.addInputPath(job, new Path(args[1]));

  FileOutputFormat.setOutputPath(job, new Path(args[2]));

  return (job.waitForCompletion(true) ? 0 : 1);

  }

}

  }

 

When executing this job on cluster user will see the following exception

java.io.IOException: wrong value class: class org.apache.hadoop.io.Text is not class org.apache.hadoop.io.LongWritable

at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:219)

at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1179)

at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1500)

at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)

at com.mr.test.MyDriver$MyDriverReducer.reduce(AppDriver.java:36)

at com.mr.test.MyDriver$MyDriverReducer.reduce(AppDriver.java:1)

at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)

at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1521) a

t org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndWrite(MapTask.java:1691)

at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1351)

at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:603)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:670)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)

at org.apache.hadoop.mapred.Child$4.run(Child.java:270)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)

at org.apache.hadoop.mapred.Child.main(Child.java:264)

Root Cause:

Combiner are mini-reducers which uses the same implementation as that of reducers. Reducer's takes input key as Text and value as LongWritable and output key as Text and value as Text

public static class MyReducer extends Reducer<Text, LongWritable, Text, Text>

In this case combiner are able to get the right input key/value and emitting output key/value as Text which are input for the Reducer and here the problem arises because in this case Reducer can only accept the key/value as Text and LongWritable.

 

Solution:

Ensure that combiners output are of same type as input type for reducers

Attachments

    Outcomes