AnsweredAssumed Answered

Hadoop 2.6.0 - MapReduce job from Hive fails with "Exception from container-launch."

Question asked by martinlocqueville on Jun 22, 2015
Latest reply on Jun 25, 2015 by martinlocqueville
I'm new to Hadoop and I've been experimenting for the past few days. I've setup Hadoop 2.6.0 on a two virtual machine cluster, one master and one slave. I've added Zookeeper 3.4.6, HBase 0.98.12.1, and Hive 0.14.0, and I've been trying to move data in form of a .csv file to Hbase through Hive.

I managed to upload the .csv into Hive through HDFS, and create the corresponding (empty) table into HBase. But whenever I try to move the data using `INSERT OVERWRITE TABLE hbase_donnees_meteo SELECT * FROM donnees_meteo WHERE id_station=1 LIMIT 10;`, the resulting MapReduce job automatically fails as soon as it starts running.

The container log only mentions that it fails to locate or load main class 408, whatever that is.

Here are the related files:

**Yarn nodemanager log**

    2015-06-18 16:35:26,025 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /opt/hadoop/hadoop-2.6.0/tmp/hadoop-hadoopuser/nm-local-dir/usercache/hadoopuser/appcache/application_1434637493524_0002/container_1434637493524_0002_01_000001/default_container_executor.sh]
    2015-06-18 16:35:26,474 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1434637493524_0002_01_000001 is : 1
    2015-06-18 16:35:26,474 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1434637493524_0002_01_000001 and exit code: 1
    ExitCodeException exitCode=1:
     at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
     at org.apache.hadoop.util.Shell.run(Shell.java:455)
     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
     at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:745)
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1434637493524_0002_01_000001
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1:
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:  at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:  at org.apache.hadoop.util.Shell.run(Shell.java:455)
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:  at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    2015-06-18 16:35:26,475 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:  at java.lang.Thread.run(Thread.java:745)
    2015-06-18 16:35:26,475 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
    2015-06-18 16:35:26,476 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1434637493524_0002_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE

**yarn-site.xml**

      <property>
        <name>yarn.resourcemanager.address</name>
        <value>hadoop-master:8032</value>
      </property>
      <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hadoop-master:8030</value>
      </property>
      <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>hadoop-master:8031</value>
      </property>
      <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>hadoop-master:8033</value>
      </property>
      <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>hadoop-master:8088</value>
      </property>
      <property>
        <name>yarn.application.classpath</name>
          <value>
          $HADOOP_CONF_DIR,
          $HADOOP_COMMON_HOME/share/hadoop/common/*,
          $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
          $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
          $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
          $YARN_HOME/share/hadoop/yarn/*
          $YARN_HOME/share/hadoop/yarn/lib/*
          $YARN_HOME/share/hadoop/mapreduce/*,
          $YARN_HOME/share/hadoop/mapreduce/lib/*
         </value>
      </property>
    </configuration>

**Hadoop webapp diagnostic**

    Application application_1434637493524_0003 failed 2 times due to AM Container for appattempt_1434637493524_0003_000002 exited with exitCode: 1
    For more detailed output, check application tracking page:http://hadoop-master:8088/proxy/application_1434637493524_0003/Then, click on links to logs of each attempt.
    Diagnostics: Exception from container-launch.
    Container id: container_1434637493524_0003_02_000001
    Exit code: 1
    Stack trace: ExitCodeException exitCode=1:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    Container exited with a non-zero exit code 1
    Failing this attempt. Failing the application.

These are the files resulting of 2 hours of searching for a fix. Please keep in mind I've only had 4 days of experience with Hadoop&co, so if I didn't include a file needed to pinpoint the error, please let me know and I'll provide it.
Also, I've tried running the example jobs provided (which means not using hive, but MR directly), and I get the same results.

Thanks in advance.

Outcomes