AnsweredAssumed Answered

HIVE: Insert hangs on FileSinkOperator: Moving tmp dir �?so slowly�?

Question asked by tony on Aug 17, 2014
only the last one thread excute filesinkoperation: Moving tmp dir: i dump this thread as below.It is too slowly, the output log is also as below:

*Thread 24452: (state = IN_JAVA)
 - org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(java.io.OutputStream, java.lang.Object, org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector, boolean, byte, boolean[]) @bci=4, line=187 (Compiled frame; information may be imprecise)
 - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(org.apache.hadoop.hive.serde2.ByteStream$Output, java.lang.Object, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector, byte[], int, org.apache.hadoop.io.Text, boolean, byte, boolean[]) @bci=80, line=486 (Compiled frame)
 - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(org.apache.hadoop.hive.serde2.ByteStream$Output, java.lang.Object, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe$SerDeParameters) @bci=29, line=439 (Compiled frame)
 - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(java.lang.Object, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector) @bci=340, line=423 (Compiled frame)
 - org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(java.lang.Object, int) @bci=241, line=566 (Compiled frame)
 - org.apache.hadoop.hive.ql.exec.Operator.forward(java.lang.Object, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector) @bci=63, line=793 (Compiled frame)
 - org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(java.lang.Object, int) @bci=13, line=45 (Compiled frame)
 - org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(java.lang.Object, java.util.Iterator, org.apache.hadoop.mapred.OutputCollector, org.apache.hadoop.mapred.Reporter) @bci=533, line=262 (Compiled frame)
 - org.apache.hadoop.mapred.ReduceTask.runOldReducer(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter, org.apache.hadoop.mapred.RawKeyValueIterator, org.apache.hadoop.io.RawComparator, java.lang.Class, java.lang.Class) @bci=165, line=502 (Interpreted frame)
 - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=552, line=450 (Interpreted frame)
 - org.apache.hadoop.mapred.Child$4.run() @bci=81, line=282 (Interpreted frame)
 - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame)
 - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame)
 - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) @bci=5, line=1117 (Interpreted frame)
 - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=897, line=271 (Interpreted frame)*

2014-08-18 16:18:09,108 INFO ExecReducer: ExecReducer: processing 1530000000 rows: used memory = 167343976
2014-08-18 16:18:12,680 INFO ExecReducer: ExecReducer: processing 1531000000 rows: used memory = 161252896
2014-08-18 16:18:16,158 INFO ExecReducer: ExecReducer: processing 1532000000 rows: used memory = 151497208
2014-08-18 16:18:19,627 INFO ExecReducer: ExecReducer: processing 1533000000 rows: used memory = 211524024
2014-08-18 16:18:23,151 INFO ExecReducer: ExecReducer: processing 1534000000 rows: used memory = 186808080
2014-08-18 16:18:26,678 INFO ExecReducer: ExecReducer: processing 1535000000 rows: used memory = 174523440
2014-08-18 16:18:30,285 INFO ExecReducer: ExecReducer: processing 1536000000 rows: used memory = 192524248
2014-08-18 16:18:33,891 INFO ExecReducer: ExecReducer: processing 1537000000 rows: used memory = 186558904
2014-08-18 16:18:37,410 INFO ExecReducer: ExecReducer: processing 1538000000 rows: used memory = 153014384
2014-08-18 16:18:40,981 INFO ExecReducer: ExecReducer: processing 1539000000 rows: used memory = 172873640
2014-08-18 16:18:44,518 INFO ExecReducer: ExecReducer: processing 1540000000 rows: used memory = 208017920
2014-08-18 16:18:48,079 INFO ExecReducer: ExecReducer: processing 1541000000 rows: used memory = 175897816
2014-08-18 16:18:51,566 INFO ExecReducer: ExecReducer: processing 1542000000 rows: used memory = 175739048
2014-08-18 16:18:55,128 INFO ExecReducer: ExecReducer: processing 1543000000 rows: used memory = 170982184
2014-08-18 16:18:58,608 INFO ExecReducer: ExecReducer: processing 1544000000 rows: used memory = 188440960
2014-08-18 16:19:02,092 INFO ExecReducer: ExecReducer: processing 1545000000 rows: used memory = 199046776
2014-08-18 16:19:05,556 INFO ExecReducer: ExecReducer: processing 1546000000 rows: used memory = 143440944
2014-08-18 16:19:09,092 INFO ExecReducer: ExecReducer: processing 1547000000 rows: used memory = 172895312
2014-08-18 16:19:13,015 INFO ExecReducer: ExecReducer: processing 1548000000 rows: used memory = 154269944
2014-08-18 16:19:16,660 INFO ExecReducer: ExecReducer: processing 1549000000 rows: used memory = 214749168
2014-08-18 16:19:20,196 INFO ExecReducer: ExecReducer: processing 1550000000 rows: used memory = 145769336
2014-08-18 16:19:23,682 INFO ExecReducer: ExecReducer: processing 1551000000 rows: used memory = 163778728
2014-08-18 16:19:27,170 INFO ExecReducer: ExecReducer: processing 1552000000 rows: used memory = 181642240
2014-08-18 16:19:30,657 INFO ExecReducer: ExecReducer: processing 1553000000 rows: used memory = 205865576
2014-08-18 16:19:34,168 INFO ExecReducer: ExecReducer: processing 1554000000 rows: used memory = 175214280
2014-08-18 16:19:37,673 INFO ExecReducer: ExecReducer: processing 1555000000 rows: used memory = 148310032
2014-08-18 16:19:41,207 INFO ExecReducer: ExecReducer: processing 1556000000 rows: used memory = 144514904
2014-08-18 16:19:44,693 INFO ExecReducer: ExecReducer: processing 1557000000 rows: used memory = 169971688
2014-08-18 16:19:48,195 INFO ExecReducer: ExecReducer: processing 1558000000 rows: used memory = 186114264

Outcomes