AnsweredAssumed Answered

HiveServer2 out of memory (GC overhead limit exceeded)

Question asked by evckumar1 on Jan 13, 2016
Latest reply on Jan 3, 2018 by pravatsutar
Hello All,

I am using Hive 1.2 with MapR 5.0, I have HS2 service started on my EDGE node with metastore thrift pointing to cluster node. I am having memory issues w.r.t HS2 from yesterday and HS2 is automatically stopping when I hit that error. Can anyone help me how to increase memory for HS2 in EDGE node? Everything is working fine from cluster side.


2016-01-12 01:50:57,806 ERROR [HiveServer2-Background-Pool: Thread-149]: ql.Driver (SessionState.java:printError(963)) - FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask. GC overhead limit exceeded
2016-01-12 01:50:57,806 INFO  [HiveServer2-Background-Pool: Thread-146]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) -
2016-01-12 01:51:04,499 WARN  [HiveServer2-Handler-Pool: Thread-19]: cli.CLIService (CLIService.java:getOperationStatus(396)) - OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=fe03f6ab-f68c-4093-98d2-8747097c9761]: The background operation was aborted
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:202)
        at org.apache.hive.service.cli.CLIService.getOperationStatus(CLIService.java:387)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.GetOperationStatus(ThriftCLIService.java:609)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetOperationStatus.getResult(TCLIService.java:1473)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$GetOperationStatus.getResult(TCLIService.java:1458)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-01-12 01:51:07,432 WARN  [HiveServer2-Background-Pool: Thread-85-SendThread(hdp0035.ch3.prod.i.com:5181)]: zookeeper.ClientCnxn (ClientCnxn.java:run(1091)) - Session 0x15142a1d7416d81 for server hdp0035.ch3.prod.i.com/10.106.128.45:5181, unexpected error, closing socket connection and attempting reconnect
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-01-12 01:51:08,553 WARN  [HiveServer2-Handler-Pool: Thread-19]: cli.CLIService (CLIService.java:getOperationStatus(396)) - OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=fe03f6ab-f68c-4093-98d2-8747097c9761]: The background operation was aborted
2016-01-12 01:51:16,844 INFO  [HiveServer2-Background-Pool: Thread-85-SendThread(hdp0037.ch3.prod.i.com:5181)]: client.ZooKeeperSaslClient (ZooKeeperSaslClient.java:run(280)) - Client will use SIMPLE-SECURITY as SASL mechanism.
2016-01-12 01:51:26,157 ERROR [HiveServer2-Background-Pool: Thread-85-SendThread(hdp0037.ch3.prod.i.com:5181)]: zookeeper.ClientCnxn (ClientCnxn.java:uncaughtException(411)) - from HiveServer2-Background-Pool: Thread-85-SendThread(hdp0037.ch3.prod.i.com:5181)
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-01-12 01:51:27,552 ERROR [Thread-6]: server.TThreadPoolServer (TThreadPoolServer.java:serve(194)) - ExecutorService threw error: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-01-12 01:51:29,110 FATAL [Thread-6]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-01-12 01:51:29,730 INFO  [Thread-4]: server.HiveServer2 (HiveServer2.java:stop(324)) - Shutting down HiveServer2
2016-01-12 01:51:30,070 INFO  [Thread-4]: thrift.ThriftCLIService (ThriftCLIService.java:stop(200)) - Thrift server has stopped
2016-01-12 01:51:30,070 INFO  [Thread-4]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2016-01-12 01:51:30,070 INFO  [Thread-4]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2016-01-12 01:51:34,317 INFO  [Thread-4]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2016-01-12 01:51:47,038 INFO  [Thread-4]: service.CompositeService (CompositeService.java:stop(104)) - Error stopping SessionManager
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-01-12 01:51:47,387 INFO  [Thread-4]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped.
2016-01-12 01:51:47,387 INFO  [Thread-4]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped.
2016-01-12 02:14:41,239 INFO  [main]: server.HiveServer2 (HiveStringUtils.java:startupShutdownMessage(691)) - STARTUP_MSG:


2016-01-12 03:32:43,383 ERROR [HiveServer2-Background-Pool: Thread-54]: mr.MapredLocalTask (MapredLocalTask.java:executeInProcess(352)) - Hive Runtime Error: Map local work exhausted memory
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.ArrayList.<init>(ArrayList.java:144)
        at org.apache.hadoop.hive.ql.exec.persistence.MapJoinEagerRowContainer.<init>(MapJoinEagerRowContainer.java:48)
        at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.process(HashTableSinkOperator.java:246)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
        at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:122)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:407)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:378)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:344)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.execute(MapredLocalTask.java:141)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
        at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
        at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
        at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2016-01-12 03:32:51,374 INFO  [HiveServer2-Background-Pool: Thread-57]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) -
2016-01-12 03:32:54,022 INFO  [HiveServer2-Background-Pool: Thread-54]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) -
2016-01-12 03:32:59,143 FATAL [Thread-6]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-01-12 03:33:03,263 INFO  [Thread-4]: server.HiveServer2 (HiveServer2.java:stop(324)) - Shutting down HiveServer2
2016-01-12 03:33:04,820 INFO  [Thread-4]: thrift.ThriftCLIService (ThriftCLIService.java:stop(200)) - Thrift server has stopped
2016-01-12 03:33:04,820 INFO  [Thread-4]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2016-01-12 03:33:04,820 INFO  [Thread-4]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2016-01-12 03:33:04,820 INFO  [Thread-4]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2016-01-12 03:33:09,502 INFO  [Thread-3]: server.HiveServer2 (HiveStringUtils.java:run(709)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at lnx0581.clt01.ulx.if.as19229.net/10.106.11.136
************************************************************/
2016-01-12 03:33:12,403 WARN  [Thread-0]: util.ShutdownHookManager (ShutdownHookManager.java:run(56)) - ShutdownHook '' failed, java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-01-12 03:33:14,984 INFO  [Thread-4]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped.
2016-01-12 03:33:15,005 INFO  [Thread-4]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped.

Outcomes