mapreduce job fails with error java.io.IOException: Mkdirs failed to create /tmp/hadoop-root/hadoop-unjar5172395916206928546/META-INF

Document created by sreedhar Employee on Feb 8, 2016
Version 1Show Document
  • View in full screen mode

Author: Sreedhar Alagonda

 

Original Publication Date: April 28, 2015

 

 

Issue:

Some times mapreduce job fails with exception like "java.io.IOException: Mkdirs failed to create /tmp/hadoop-root/xxxx"

 

A simple wordcount  example failing  as follows..

 

Wordcount job:  
hadoop jar /opt/mapr/hadoop/hadoop-0.20.2/hadoop-0.20.2-dev-examples.jar wordcount /in/data.txt  /out

OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file:

  /tmp/hsperfdata_root/7519

Try using the -Djava.io.tmpdir= option to select an alternate temp location.

 

Exception in thread "main" java.io.IOException: Mkdirs failed to create /tmp/hadoop-root/hadoop-unjar5172395916206928546/META-INF

at org.apache.hadoop.util.RunJar.ensureDirectory(RunJar.java:111)

at org.apache.hadoop.util.RunJar.unJar(RunJar.java:87)

at org.apache.hadoop.util.RunJar.unJar(RunJar.java:64)

at org.apache.hadoop.util.RunJar.main(RunJar.java:188)

Environment:

 

All MaR versions

 

Solution:

 

One of the symptom for this issue points to space issue in /tmp. Just do df -h and see /tmp mount point having sufficient space.

 

[root@node1 crash]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/vg_node1-lv_root

  19G 19G 0 100% /

tmpfs 3.9G 0 3.9G 0% /dev/shm

/dev/sda1 477M 33M 419M 8% /boot

Figure out what is taking  large space and clean it up..

 

root@node1 /]# du - sch *

15G tmp

1G usr

1G var

...........

........

19G total

/tmp holding lot of space   so clean it up ...

 

Run wordcount job now, it will launch job..

 

5/03/29 14:19:30 INFO fs.JobTrackerWatcher: Current running JobTracker is: node2/10.10.80.221:9001

2015-03-29 14:19:30,7618 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:1290 Thread: 26124 Lookup of volume mapr.var failed, error Connection reset by peer(104), CLDB: 10.10.80.221:7222 backing off ...

15/03/29 14:19:31 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

15/03/29 14:19:32 INFO input.FileInputFormat: Total input paths to process : 1

15/03/29 14:19:32 INFO mapred.JobClient: Creating job's output directory at /out

15/03/29 14:19:32 INFO mapred.JobClient: Creating job's user history location directory at /out/_logs

15/03/29 14:19:32 INFO Configuration.deprecation: topology.script.number.args is deprecated. Instead, use net.topology.script.number.args

15/03/29 14:19:32 INFO mapred.JobClient: root (auth:SIMPLE), realuser: null

15/03/29 14:19:33 INFO mapred.JobClient: Running job: job_201504271559_0001

15/03/29 14:19:34 INFO mapred.JobClient: map 0% reduce 0%

Attachments

    Outcomes