AnsweredAssumed Answered

hive error with large input data

Question asked by anu on Jul 26, 2012
Latest reply on Jul 26, 2012 by anu
We have set up a mapr cluster with 7 nodes having Intel(R) Xeon(R) CPU E5606  @ 2.13GHz CPU and 8 GB RAM.
We are getting the following exection while running the following command

    hive -e "use $database;create table sz_grp_4_data as select week,store,item_id,value,lvl1,lvl2,lvl3,lvl4,lvl5,lvl6,lvl7,lvl8,region,store_sz,store_acv from table_final where store_szgrp=**4**";

    select count(*) from table_final where store_szgrp=**4**";  -- got output  "**1294168482**"

**The below code worked fine**

    hive -e "use $database;create table sz_grp_4_data as select week,store,item_id,value,lvl1,lvl2,lvl3,lvl4,lvl5,lvl6,lvl7,lvl8,region,store_sz,store_acv from table_final where store_szgrp=**1**";

    select count(*) from table_final where store_szgrp=1";  -- got output  "**647981173**"

While running the first query we got the following exception

    java.lang.RuntimeException: Hive Runtime Error while closing operators
    at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:226)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:405)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:336)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
    at org.apache.hadoop.mapred.Child.main(Child.java:264)
    **Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output to: maprfs://10.132.213.31:7222/tmp/hive-root/hive_2012-07-25_17-17-10_080_8302669910304016669/_tmp.-ext-10002/000601_0**

    java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row  

    {"week":6,"store":821446,"item_id":23132613006,"value":0.0,"lvl1":2,"lvl2":3,"lvl3":1,"lvl4":3,"lvl5":2,"lvl6":6,"lvl7":1,"lvl8":3,"region":8,"store_sz":910.0,"store_acv":3072352.0,"store_szgrp":2}
       at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:405)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:336)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
      at org.apache.hadoop.mapred.Child.main(Child.java:264)
        Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"week":6,"store":821446,"item_id":23132613006,"value":0.0,"lvl1":2,"lvl2":3,"lvl3":1,"lvl4":3,"lvl5":2,"lvl6":6,"lvl7":1,"lvl8":3,"region":8,"store_sz":910.0,"store_acv":3072352.0,"store_szgrp":2}
          at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
          at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
          ... 8 more
          **Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Create failed for file: /tmp/hive-root/hive_2012-07-25_17-17-10_080_8302669910304016669/_tmp.-ext-10002/_tmp.000601_1, error: No space left on device (28)**
          at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:236)
          at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:461)
          at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:507)
          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
          at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
          at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
          at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
          at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:87)
          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
          at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
          at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:87)
          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
          at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
          at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:78)
          at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
          at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
          at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:527)
      ... 9 more
       Caused by: java.io.IOException: Create failed for file: /tmp/hive-**root/hive_2012-07-25_17-17-10_080_8302669910304016669/_tmp.-ext-10002/_tmp.000601_1, error: No space left on device (28)**
       at com.mapr.fs.MapRClient.create(MapRClient.java:176)



Outcomes