AnsweredAssumed Answered

Unable to configure MapR local volume as the spill directory for Apache Drill

Question asked by aalvarez on Jul 28, 2016
Latest reply on Aug 1, 2016 by aalvarez

Hi,

I was getting some memory allocation problems and decided to configure MapR local volume as the spill directory. However, I am now having problems identifying what failed after I change the configurations.

 

The current error looks this:

0: jdbc:drill:> select * from dfs.views.query limit 2;

Error: SYSTEM ERROR: JaninoRuntimeException: Code attribute in class "org.apache.drill.exec.test.generated.HashJoinProbeGen283" grows beyond 64 KB

 

Fragment 3:13

 

[Error Id: 4f8b2248-a937-42dc-8961-961aa6835397 on metrics4:31010] (state=,code=0)

 

Where:

     1) The query that threw this output used to work before I configured the spill directory, so not the source of the problem. No other queries works. (Updated: some less complex queries still work)

     2) Drill is failing every single time on this machine: metrics4

     3) this is what the drillbit.log looks like in that machine:

 

2016-07-28 21:05:40,317 [286555fd-48ed-0b9e-04d2-32048ddc9517:frag:1:17] INFO  o.a.d.e.w.f.FragmentStatusReporter - 286555fd-48ed-0b9e-04d2-32048ddc9517:1:17: State to report: CANCELLED

2016-07-28 21:05:40,317 [drill-executor-45] WARN  o.a.d.exec.rpc.control.WorkEventBus - Fragment 286555fd-48ed-0b9e-04d2-32048ddc9517:1:17 not found in the work bus.

2016-07-28 21:05:40,326 [286555fd-48ed-0b9e-04d2-32048ddc9517:frag:1:25] INFO  o.a.d.e.w.fragment.FragmentExecutor - 286555fd-48ed-0b9e-04d2-32048ddc9517:1:25: State change requested CANCELLATION_REQUESTED --> FINISHED

2016-07-28 21:05:40,326 [286555fd-48ed-0b9e-04d2-32048ddc9517:frag:1:25] INFO  o.a.d.e.w.f.FragmentStatusReporter - 286555fd-48ed-0b9e-04d2-32048ddc9517:1:25: State to report: CANCELLED

2016-07-28 21:05:40,326 [drill-executor-47] WARN  o.a.d.exec.rpc.control.WorkEventBus - Fragment 286555fd-48ed-0b9e-04d2-32048ddc9517:1:25 not found in the work bus.

 

Here is how I configure MapR local volume as the spill directory for Apache Drill based on this website:

1. Create a MapR local volume for each node through MCS:

Untitled-2.png

     After installed, volumes were active and no alarms were triggered. Masked text is real name of nodes.

2. Add an environment variable to drill-env.sh on all nodes.

export DRILL_LOCALHOST=`hostname -f`

 

3. Add the configurations for spill directory in drill-override.conf on all nodes.*

drill.exec: {

  cluster-id: "com_mapr_corp_metrics-drillbits",

  zk.connect: "metrics1:5181,metrics2:5181,metrics3:5181"

  sort.external.spill.directories: ["/tmp/"${DRILL_LOCALHOST}],

  sort.external.spill.fs: "maprfs:///"

}

 

*Only have zookeeper installed in metrics1, metrics2 & metrics3, hence why m4cis not reflected there.

4. Restart all drillbits

1

 

 

maprcli node services -name drill-bits -action restart -filter csvc=="drill-bits"

 

5. Check the configurations

0: jdbc:drill:> select * from sys.drillbits where `current`=true;

+-----------------------------+------------+---------------+------------+----------+

|          hostname           | user_port  | control_port  | data_port  | current  |

+-----------------------------+------------+---------------+------------+----------+

| metrics3  | 31010      | 31011         | 31012      | true     |

+-----------------------------+------------+---------------+------------+----------+

1 row selected (0.146 seconds)

 

0: jdbc:drill:> select name,string_val from sys.boot where name in ('drill.exec.sort.external.spill.fs','drill.exec.sort.external.spill.directories');

+---------------------------------------------+-----------------------------------------------------------------------------------------------------------+

|                    name                     |                                                string_val                                                 |

+---------------------------------------------+-----------------------------------------------------------------------------------------------------------+

| drill.exec.sort.external.spill.directories  | [

    # merge of drill-override.conf: 25,env var DRILL_LOCALHOST

    "/tmp/metrics3"

]  |

| drill.exec.sort.external.spill.fs           | "maprfs:///"                                                                                              |

+---------------------------------------------+-----------------------------------------------------------------------------------------------------------+

2 rows selected (0.253 seconds)

 

I tried reverting that node alone to the previous state but it didn't work. Ran out of ideas.

Any help or ideas will be welcomed. I can't find any documentation on the current error.

 

 

Environment: mapr 5.1 drill 1.6 on CentOS 7.2

Outcomes