AnsweredAssumed Answered

DrillBits Service Always Crashes in MapR Cluster

Question asked by PETER.EDIKE on Jan 29, 2018
Latest reply on Feb 9, 2018 by maprcommunity

Hello everyone,  

Please I have a problem with my MapR Cluster, I had set it up and ran a job which completed successfully, however I am trying to use drill bits to query the data and i continually get a 503 Service Unavailable  message.  I check MCS and I see that the Drill Bits Service is down. I try to start it, which never succeeds. I go the DrillBit Logs and I see the following exception 

 

Distribution Specific Configuration File: file:/opt/mapr/drill/drill-1.11.0/conf/drill-distrib.conf

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This file is empty by default. Default Drill environment settings appear
# in drill-config.sh. Distributions can replace this file with a
# distribution-specific version that sets environment variables and options
# specific to that distribution. Users should not put anything in this file;
# put user options in drill-env.sh instead.

# MapR-specific configuration for Drill

drill.exec: {
cluster-id: "test.cluster.com-drillbits",
zk.connect: "172.25.10.86:5181",
rpc.user.client.threads: "4",
}

Override File: file:/opt/mapr/drill/drill-1.11.0/conf/drill-override.conf

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This file tells Drill to consider this module when class path scanning.
# This file can also include any supplementary configuration information.
# This file is in HOCON format, see https://github.com/typesafehub/config/blob/master/HOCON.md for more information.

# See 'drill-override-example.conf' for example configurations

drill.exec: {
}


2018-01-26 19:55:49,366 [main] INFO o.a.d.common.scanner.BuildTimeScan - Loaded prescanned packages [org.apache.drill.storage, org.apache.drill.exec.store.mock, org.apache.drill.common.logical, org.apache.dril$
2018-01-26 19:55:50,628 [main] INFO o.a.d.c.scanner.ClassPathScanner - Scanning packages [org.apache.drill.exec.store.jdbc, org.apache.drill.exec.rpc.security.maprsasl, org.apache.drill.exec.store.mongo, org.a$
2018-01-26 19:55:53,275 [main] INFO o.a.d.e.s.s.PersistentStoreRegistry - Using the configured PStoreProvider class: 'org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider'.
2018-01-26 19:55:53,677 [main] INFO o.apache.drill.exec.server.Drillbit - Construction completed (3043 ms).
2018-01-26 19:55:53,844 [main] INFO o.a.d.c.s.persistence.ScanResult - loading 16 classes for org.apache.drill.common.logical.data.LogicalOperator took 8ms
2018-01-26 19:55:53,851 [main] INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for org.apache.drill.common.logical.StoragePluginConfig took 5ms
2018-01-26 19:55:53,857 [main] INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for org.apache.drill.common.logical.FormatPluginConfig took 4ms
2018-01-26 19:55:53,995 [main] INFO o.a.d.c.s.persistence.ScanResult - loading 71 classes for org.apache.drill.exec.physical.base.PhysicalOperator took 75ms
2018-01-26 19:55:54,097 [main] INFO o.a.d.c.s.persistence.ScanResult - loading 40 classes for org.apache.drill.exec.physical.impl.BatchCreator took 33ms
2018-01-26 19:55:54,104 [main] INFO o.a.d.c.s.persistence.ScanResult - loading 6 classes for org.apache.drill.exec.physical.impl.RootCreator took 4ms
2018-01-26 19:55:54,702 [main] INFO o.a.d.c.s.persistence.ScanResult - loading 1 classes for org.apache.drill.exec.expr.fn.PluggableFunctionRegistry took 4ms
2018-01-26 19:55:55,645 [main] INFO o.a.d.c.scanner.ClassPathScanner - Scanning packages [org.apache.drill.exec.store.jdbc, org.apache.drill.exec.rpc.security.maprsasl, org.apache.drill.exec.store.mongo, org.a$
2018-01-26 19:55:55,733 [main] INFO o.a.d.c.s.persistence.ScanResult - loading 114 classes for org.apache.hadoop.hive.ql.udf.generic.GenericUDF took 83ms
2018-01-26 19:55:56,182 [main] INFO o.a.d.c.s.persistence.ScanResult - loading 68 classes for org.apache.hadoop.hive.ql.exec.UDF took 168ms
2018-01-26 19:55:56,209 [main] INFO o.a.d.e.e.f.FunctionImplementationRegistry - Function registry loaded. 450 functions loaded in 2091 ms.
2018-01-26 19:55:56,220 [main] INFO o.a.d.e.e.f.FunctionImplementationRegistry - Created and validated local udf directory [/tmp/drill/udf/udf/local]
2018-01-26 19:55:56,224 [main] INFO o.a.drill.exec.compile.CodeCompiler - Plain java code generation preferred: false
2018-01-26 19:55:56,420 [main] INFO o.a.d.c.s.persistence.ScanResult - loading 9 classes for org.apache.drill.exec.store.StoragePlugin took 17ms
2018-01-26 19:55:56,424 [main] INFO o.a.d.e.s.StoragePluginRegistryImpl - No storage plugin instances configured in persistent store, loading bootstrap configuration.
2018-01-26 19:55:56,425 [main] INFO o.a.d.e.s.StoragePluginRegistryImpl - Loading the storage plugin configs from URLs [jar:file:/opt/mapr/drill/drill-1.11.0/jars/drill-mongo-storage-1.11.0-mapr.jar!/bootstrap$
2018-01-26 19:55:56,572 [main] ERROR o.apache.drill.exec.server.Drillbit - Failure during initial startup of Drillbit.
org.apache.drill.common.exceptions.DrillRuntimeException: unable to put
at org.apache.drill.exec.coord.zk.ZookeeperClient.putIfAbsent(ZookeeperClient.java:326) ~[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:118) ~[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at org.apache.drill.exec.store.StoragePluginRegistryImpl.definePluginConfig(StoragePluginRegistryImpl.java:204) ~[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at org.apache.drill.exec.store.StoragePluginRegistryImpl.createPlugins(StoragePluginRegistryImpl.java:149) ~[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at org.apache.drill.exec.store.StoragePluginRegistryImpl.init(StoragePluginRegistryImpl.java:128) ~[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:164) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:349) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:323) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:319) [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /drill/sys.storage_plugins/mongo
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113) ~[zookeeper-3.4.5-mapr-1503.jar:3.4.5-mapr-1503--1]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.5-mapr-1503.jar:3.4.5-mapr-1503--1]
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[zookeeper-3.4.5-mapr-1503.jar:3.4.5-mapr-1503--1]
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691) ~[curator-framework-2.7.1.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675) ~[curator-framework-2.7.1.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.7.1.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:672) ~[curator-framework-2.7.1.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453) ~[curator-framework-2.7.1.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443) ~[curator-framework-2.7.1.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) ~[curator-framework-2.7.1.jar:na]
at org.apache.drill.exec.coord.zk.ZookeeperClient.putIfAbsent(ZookeeperClient.java:318) ~[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
... 8 common frames omitted


Please I do not know what the problem is, some pointers will be most helpful

 

Thank you 

Outcomes