Impala crashes because of too many RA threads

Document created by Hao Zhu Employee on Feb 7, 2016
Version 1Show Document
  • View in full screen mode

Author: Hao Zhu

 

Original Publication Date: January 20, 2016

 

Environment :

Impala 1.4.1

 

Symptom:

Impala crashes with below stacktrace in impalad core file, and the size of core file is huge.

(gdb) bt

#0 0x0000003952432925 in raise () from /lib64/libc.so.6

#1 0x000000395243408d in abort () from /lib64/libc.so.6

#2 0x000000000145b389 in tcmalloc::Log (mode=tcmalloc::kCrash, filename=<value optimized out>, line=<value optimized out>, a=...

  at src/internal_logging.cc:119

#3 0x0000000001460d37 in New (p=123610, len=1) at src/page_heap_allocator.h:73

#4 tcmalloc::NewSpan (p=123610, len=1) at src/span.cc:54

#5 0x00000000014617a0 in tcmalloc::PageHeap::Carve (this=0x38db000, span=0x7f7a83792fb0, n=1) at src/page_heap.cc:174

#6 0x00000000014621df in tcmalloc::PageHeap::New (this=0x38db000, n=1) at src/page_heap.cc:97

#7 0x000000000145844d in tcmalloc::CentralFreeList::Populate (this=0x1bd2520) at src/central_freelist.cc:318

#8 0x0000000001458658 in tcmalloc::CentralFreeList::FetchFromSpansSafe (this=0x1bd2520) at src/central_freelist.cc:285

#9 0x00000000014586d8 in tcmalloc::CentralFreeList::RemoveRange (this=0x1bd2520, start=0x7f5d8c963238, end=0x7f5d8c963230, N=32)

#10 0x000000000145a4e4 in tcmalloc::ThreadCache::FetchFromCentralCache (this=0xda42c00, cl=<value optimized out>, byte_size=352)

#11 0x0000000001471fc8 in Allocate (size=336) at src/thread_cache.h:341

#12 do_malloc (size=336) at src/tcmalloc.cc:1068

#13 cpp_alloc (size=336) at src/tcmalloc.cc:1354

#14 tc_new (size=336) at src/tcmalloc.cc:1530

#15 0x0000000000ab2a6b in impala::HdfsScanNode::AllocateScanRange (this=0x387cb600, file=0xbfeb758 "maprfs:/edw_data/hp_tab/call_

  len=67108864, offset=62545461248, partition_id=<value optimized out>, disk_id=0, try_cache=false)

  at /root/builds/opensource/impala-1.4.1/dl/mapr-impala-1.4.1/be/src/exec/hdfs-scan-node.cc:206

#16 0x0000000000ab74e7 in impala::HdfsScanNode::Prepare (this=0x387cb600, state=0xc6d9c00)

  at /root/builds/opensource/impala-1.4.1/dl/mapr-impala-1.4.1/be/src/exec/hdfs-scan-node.cc:455

#17 0x0000000000a9de20 in impala::ExecNode::Prepare (this=0x30de83c0, state=0xc6d9c00) at /root/builds/opensource/impala-1.4.1/dl

#18 0x0000000000b0ece0 in impala::AggregationNode::Prepare (this=0x30de83c0, state=0xc6d9c00)

  at /root/builds/opensource/impala-1.4.1/dl/mapr-impala-1.4.1/be/src/exec/aggregation-node.cc:84

#19 0x0000000000a80105 in impala::PlanFragmentExecutor::Prepare (this=0x376a61b0, request=...)

  at /root/builds/opensource/impala-1.4.1/dl/mapr-impala-1.4.1/be/src/runtime/plan-fragment-executor.cc:213

#20 0x000000000092f393 in impala::ImpalaServer::FragmentExecState::Prepare (this=0x376a6000, exec_params=...)

  at /root/builds/opensource/impala-1.4.1/dl/mapr-impala-1.4.1/be/src/service/fragment-exec-state.cc:43

#21 0x00000000008d675a in impala::ImpalaServer::StartPlanFragmentExecution (this=0x39dfb80, exec_params=...)

  at /root/builds/opensource/impala-1.4.1/dl/mapr-impala-1.4.1/be/src/service/impala-server.cc:1115

#22 0x00000000008d7132 in impala::ImpalaServer::ExecPlanFragment (this=0x39dfb80, return_val=..., params=...)

  at /root/builds/opensource/impala-1.4.1/dl/mapr-impala-1.4.1/be/src/service/impala-server.cc:1027

#23 0x00000000009fbd1d in impala::ImpalaInternalServiceProcessor::process_ExecPlanFragment (this=0xa9de6c0, seqid=0, iprot=<value

  callContext=<value optimized out>) at /root/builds/opensource/impala-1.4.1/dl/mapr-impala-1.4.1/b

Root Cause:

With high concurrency of queries, Impala on MapR-FS may spawn too many RA(ReadAhead) threads and ask for huge amount of virtual memory from OS. Then impalad crashes when "tcmalloc" fails.

Solution:

1. Apply the latest MapR EBF patch for MapR 4.0.1 or 4.0.2.

The latest EBF patch for 4.0.1 and 4.0.2 can be found here:

http://archive.mapr.com/patches/releases/v4.0.1/

http://archive.mapr.com/patches/releases/v4.0.2/

 

Note: This fix is already in MapR 4.1GA or above version.

If you are using 4.1 or above MapR version, please skip this step and continue with below steps.

 

2. Change impala file client from C++ Hadoop client to JAVA Hadoop client.

Please follow this article:

How to switch Impala file client from MapR C++ Client to Hadoop JAVA Client

 

3. Set property MAPR_IMPALA_RA_THROTTLE to enable the RA thread throttle.

The parameter should be set in /opt/mapr/impala/impala-1.4.1/bin/impalaserver on all impala nodes.

export JAVA_TOOL_OPTIONS+=" -Dmapr.library.flatclass -DMAPR_IMPALA_RA_THROTTLE"

 

4. Restart impala cluster

Attachments

    Outcomes