AnsweredAssumed Answered

I am not able to Query in Impala after enabling Kerberos in MapR 5.2 on centOS 7.3

Question asked by vik686 on Dec 29, 2017
Latest reply on Jan 2, 2018 by cathy

I am not able to Query in Impala after enabling Kerberos in MapR 5.2 on centOS 7.3

 

I have a 5 node cluster:

 

edge.localdomain

master.localdomain - catalogd, statestore, hiveserver2

node1.localdomain - impala demon

node2.localdomain - impala demon 

node3.localdomain - impala demon

 

This is the error I am getting while querying in Impala:

 

[user1@master ~]$ impala-shell -s mapr -i node1 -k
Starting Impala Shell using Kerberos authentication
Using service name 'mapr'
Connected to node1:21000
Server version: impalad version 2.7.0 RELEASE (build 801f0c60a7e164faa59b959348d69d1b7e2728c4)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.7.0 (801f0c6) built on Wed Oct 25 21:38:31 UTC 2017)

To see more tips, run the TIP command.
***********************************************************************************
[node1:21000] > invalidate metadata;
Query: invalidate metadata
Query submitted at: 2017-12-29 05:03:05 (Coordinator: http://node1.localdomain:25000)
ERROR: AnalysisException: This Impala daemon is not ready to accept user requests. Status: Waiting for catalog update from the StateStore.

[node1:21000] >

 

Error in catalogd.master.localdomain.mapr.log.INFO.***

 

Log file created at: 2017/12/29 05:01:51
Running on machine: master.localdomain
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1229 05:01:51.315654 2846 logging.cc:120] stdout will be logged to this file.
E1229 05:01:51.315883 2846 logging.cc:121] stderr will be logged to this file.
I1229 05:01:51.316455 2846 minidump.cc:222] Setting minidump size limit to 20971520.
I1229 05:01:51.331010 2846 authentication.cc:682] Using internal kerberos principal "mapr/master.localdomain@EXAMPLE.COM"
I1229 05:01:51.331029 2846 authentication.cc:1018] Internal communication is authenticated with Kerberos
I1229 05:01:51.331250 2932 authentication.cc:498] Registering mapr/master.localdomain@EXAMPLE.COM, keytab file /opt/mapr/conf/mapr.keytab
I1229 05:01:51.332458 2846 authentication.cc:802] Waiting for Kerberos ticket for principal: mapr/master.localdomain@EXAMPLE.COM
I1229 05:01:51.349499 2846 authentication.cc:804] Kerberos ticket granted to mapr/master.localdomain@EXAMPLE.COM
I1229 05:01:51.349548 2846 authentication.cc:682] Using external kerberos principal "mapr/master.localdomain@EXAMPLE.COM"
I1229 05:01:51.349558 2846 authentication.cc:1034] External communication is authenticated with Kerberos
I1229 05:01:51.349795 2846 init.cc:204] catalogd version 2.7.0 RELEASE (build 801f0c60a7e164faa59b959348d69d1b7e2728c4)
Built on Wed Oct 25 21:38:31 UTC 2017
I1229 05:01:51.349803 2846 init.cc:205] Using hostname: master.localdomain
I1229 05:01:51.350653 2846 logging.cc:156] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=false
--num_metadata_loading_threads=16
--sentry_config=
--asm_module_dir=
--disable_optimization_passes=false
--dump_ir=false
--opt_module_dir=
--perf_map=false
--print_llvm_ir_instruction_count=false
--unopt_module_dir=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_kudu=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=master.localdomain
--keytab_file=/opt/mapr/conf/mapr.keytab
--krb5_conf=
--krb5_debug_file=
--load_auth_to_local_rules=false
--max_minidumps=9
--mem_limit=80%
--minidump_path=/opt/mapr/impala/impala-2.7.0/logs/minidumps/catalogd
--minidump_size_limit_hint_kb=20480
--principal=mapr/master.localdomain@EXAMPLE.COM
--redaction_rules_file=
--max_log_files=10
--pause_monitor_sleep_time_ms=500
--pause_monitor_warn_threshold_ms=10000
--log_filename=catalogd
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--enable_quadratic_probing=true
--skip_lzo_version_check=false
--parquet_min_filter_reject_ratio=0.10000000000000001
--max_row_batches=0
--runtime_filter_wait_time_ms=1000
--suppress_unknown_disk_id_warnings=false
--kudu_max_row_batches=0
--kudu_scanner_keep_alive_period_us=15000000
--kudu_scanner_keep_alive_period_sec=15
--kudu_scanner_timeout_sec=60
--pick_only_leaders_for_tests=false
--kudu_session_timeout_seconds=60
--convert_legacy_hive_parquet_utc_timestamps=false
--max_page_header_size=8388608
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--internal_principals_whitelist=hdfs
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--datastream_sender_timeout_ms=120000
--max_cached_file_handles=0
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_remote_hdfs_io_threads=8
--num_s3_io_threads=16
--num_threads_per_disk=0
--read_size=8388608
--backend_client_connection_num_retries=3
--backend_client_rpc_timeout_ms=300000
--catalog_client_connection_num_retries=3
--catalog_client_rpc_timeout_ms=0
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--coordinator_rpc_threads=12
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=master.localdomain
--state_store_subscriber_port=23020
--use_statestore=true
--s3a_access_key_cmd=
--s3a_secret_key_cmd=
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--max_filter_error_rate=0.75
--num_threads_per_core=3
--use_local_tz_for_unix_timestamp_conversions=false
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--max_vcore_oversubscription_ratio=2.5
--rm_mem_expansion_timeout_ms=5000
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--default_pool_max_queued=200
--default_pool_max_requests=-1
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--require_username=false
--disable_admission_control=false
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--authorized_proxy_user_config_delimiter=,
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--abort_on_failed_lineage_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--lineage_event_log_dir=
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_lineage_log_file_size=5000
--max_profile_log_file_size=5000
--max_profile_log_files=10
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_private_key_password_cmd=
--ssl_server_certificate=
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_heartbeat_tcp_timeout_seconds=3
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--statestore_update_tcp_timeout_seconds=300
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/mapr/impala/impala-2.7.0
--webserver_interface=
--webserver_password_file=
--webserver_port=25020
--webserver_private_key_file=
--webserver_private_key_password_cmd=
--webserver_x_frame_options=DENY
--flagfile=
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/opt/mapr/impala/impala-2.7.0/logs
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=1800
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=0
--vmodule=
I1229 05:01:51.350775 2846 init.cc:210] Cpu Info:
Model: Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Cores: 2
L1 Cache: 32.00 KB (Line: 64.00 B)
L2 Cache: 256.00 KB (Line: 64.00 B)
L3 Cache: 20.00 MB (Line: 64.00 B)
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
I1229 05:01:51.350786 2846 init.cc:211] Disk Info:
Num disks 6:
fd (rotational=true)
sda (rotational=true)
sdb (rotational=true)
sdc (rotational=true)
sr (rotational=2017-12-29 05:01:54,431 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes whe
re applicable
2017-12-29 05:01:55,119 INFO [main] security.ClusterServerTicketGeneration (ClusterServerTicketGeneration.java:generateTicketAndSetServerKey(187)) - Server key was cached for cluster: my.cluster.com
true)
dm- (rotational=true)

I1229 05:01:51.350806 2846 init.cc:212] Physical Memory: 15.51 GB
I1229 05:01:51.350813 2846 init.cc:213] OS version: Linux version 3.10.0-514.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Nov 22 16:42:41
UTC 2016
Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC
I1229 05:01:51.350816 2846 init.cc:214] Process ID: 2846
I1229 05:01:55.239585 2846 webserver.cc:233] Starting webserver on 0.0.0.0:25020
I1229 05:01:55.239614 2846 webserver.cc:247] Document root: /opt/mapr/impala/impala-2.7.0
I1229 05:01:55.239796 2846 webserver.cc:331] Webserver started
I1229 05:01:55.547180 2846 GlogAppender.java:126] Logging initialized. Impala: INFO, All other: INFO
I1229 05:01:55.548128 2846 JniCatalog.java:98] Java Version Info: OpenJDK Runtime Environment (1.8.0_151-b12)
I1229 05:01:55.573961 2846 HiveConf.java:181] Found configuration file file:/opt/mapr/impala/impala-2.7.0/conf/hive-site.xml
I1229 05:01:56.515225 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:56.548943 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 1
W1229 05:01:56.667676 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1647)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:71)
at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:138)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:173)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:100)
I1229 05:01:56.668632 2846 HiveMetaStoreClient.java:530] Connected to metastore.
I1229 05:01:56.985664 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:56.986095 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 2
W1229 05:01:57.010318 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1647)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:71)
at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:138)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:173)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:100)
I1229 05:01:57.013741 2846 HiveMetaStoreClient.java:530] Connected to metastore.
I1229 05:01:57.014441 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:57.015017 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 3
W1229 05:01:57.041110 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1647)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:71)
at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:138)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:173)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:100)
I1229 05:01:57.043483 2846 HiveMetaStoreClient.java:530] Connected to metastore.
I1229 05:01:57.043954 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:57.044397 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 4
W1229 05:01:57.069612 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1647)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:71)
at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:138)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:173)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:100)
I1229 05:01:57.070348 2846 HiveMetaStoreClient.java:530] Connected to metastore.
I1229 05:01:57.070868 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:57.071313 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 5
W1229 05:01:57.090412 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1647)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:71)
at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:138)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:173)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:100)
I1229 05:01:57.091586 2846 HiveMetaStoreClient.java:530] Connected to metastore.
I1229 05:01:57.092075 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:57.092514 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 6
W1229 05:01:57.248075 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1647)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:71)
at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:138)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:173)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:100)
I1229 05:01:57.248416 2846 HiveMetaStoreClient.java:530] Connected to metastore.
I1229 05:01:57.248888 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:57.249280 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 7
W1229 05:01:57.268987 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1647)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:71)
at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:138)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:173)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:100)
I1229 05:01:57.272881 2846 HiveMetaStoreClient.java:530] Connected to metastore.
I1229 05:01:57.273326 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:57.273733 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 8
W1229 05:01:57.295132 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1647)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:71)
at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:138)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:173)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:100)
I1229 05:01:57.306717 2846 HiveMetaStoreClient.java:530] Connected to metastore.
I1229 05:01:57.307155 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:57.307564 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 9
W1229 05:01:57.335353 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1647)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:71)
at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:138)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:173)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:100)
I1229 05:01:57.363628 2846 HiveMetaStoreClient.java:530] Connected to metastore.
I1229 05:01:57.364044 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:57.364439 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 10
W1229 05:01:57.623114 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1647)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
at com.cloudera.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:71)
at com.cloudera.impala.catalog.MetaStoreClientPool.addClients(MetaStoreClientPool.java:138)
at com.cloudera.impala.catalog.Catalog.<init>(Catalog.java:97)
at com.cloudera.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:173)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:100)
I1229 05:01:57.623441 2846 HiveMetaStoreClient.java:530] Connected to metastore.
E1229 05:01:58.533435 2846 MetaStoreUtils.java:1335] Got exception: org.apache.thrift.transport.TTransportException null
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_databases(ThriftHiveMetastore.java:847)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_databases(ThriftHiveMetastore.java:835)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1148)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy4.getAllDatabases(Unknown Source)
at com.cloudera.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:639)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:103)
E1229 05:01:58.554502 2846 MetaStoreUtils.java:1336] Converting exception to MetaException
W1229 05:01:58.570762 2846 RetryingMetaStoreClient.java:199] MetaStoreClient lost connection. Attempting to reconnect.
Java exception follows:
MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1337)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy4.getAllDatabases(Unknown Source)
at com.cloudera.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:639)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:103)
W1229 05:01:59.579727 2846 TIOStreamTransport.java:112] Error closing output stream.
Java exception follows:
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
at org.apache.thrift.transport.TSocket.close(TSocket.java:235)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:558)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:373)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:146)
at com.sun.proxy.$Proxy4.getAllDatabases(Unknown Source)
at com.cloudera.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:639)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:103)
I1229 05:01:59.580054 2846 HiveMetaStoreClient.java:559] Closed a connection to metastore, current connections: 9
I1229 05:01:59.580253 2846 HiveMetaStoreClient.java:433] Trying to connect to metastore with URI thrift://master.localdomain:9083
I1229 05:01:59.580641 2846 HiveMetaStoreClient.java:478] Opened a connection to metastore, current connections: 10
W1229 05:01:59.598902 2846 HiveMetaStoreClient.java:502] set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4118)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4104)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:494)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:378)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:146)
at com.sun.proxy.$Proxy4.getAllDatabases(Unknown Source)
at com.cloudera.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:639)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:103)
I1229 05:01:59.599505 2846 HiveMetaStoreClient.java:530] Connected to metastore.
E1229 05:01:59.599967 2846 MetaStoreUtils.java:1335] Got exception: org.apache.thrift.transport.TTransportException null
Java exception follows:
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_databases(ThriftHiveMetastore.java:847)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_databases(ThriftHiveMetastore.java:835)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1148)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy4.getAllDatabases(Unknown Source)
at com.cloudera.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:639)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:103)
E1229 05:01:59.600085 2846 MetaStoreUtils.java:1336] Converting exception to MetaException
E1229 05:01:59.600625 2846 CatalogServiceCatalog.java:679] MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
E1229 05:01:59.601193 2846 JniCatalog.java:105] Error initialializing Catalog. Please run 'invalidate metadata'
Java exception follows:
com.cloudera.impala.catalog.CatalogException: Error initializing Catalog. Catalog may be empty.
at com.cloudera.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:680)
at com.cloudera.impala.service.JniCatalog.<init>(JniCatalog.java:103)
Caused by: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1337)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy4.getAllDatabases(Unknown Source)
at com.cloudera.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:639)
... 1 more
I1229 05:01:59.613572 2846 statestore-subscriber.cc:185] Starting statestore subscriber
I1229 05:01:59.615403 2846 thrift-server.cc:434] ThriftServer 'StatestoreSubscriber' started on port: 23020
I1229 05:01:59.615417 2846 statestore-subscriber.cc:203] Registering with statestore
I1229 05:01:59.634295 2846 statestore-subscriber.cc:207] statestore registration successful
I1229 05:01:59.635524 2846 thrift-server.cc:434] ThriftServer 'CatalogService' started on port: 26000
I1229 05:01:59.635534 2846 catalogd-main.cc:96] CatalogService started on port: 26000
I1229 05:01:59.644124 3790 catalog-server.cc:232] Catalog Version: 0 Last Catalog Version: 0
I1229 05:11:59.784351 3790 catalog-server.cc:232] Catalog Version: 0 Last Catalog Version: 0
[root@master logs]#

 

I didn't found any kerberos related issues in logs.

 

Following were the additional configurations I added in /opt/mapr/impala/impala-2.7.0/conf/env.sh, 

 

master.localdomain

 

IMPALA_SERVER_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore \
-authorized_proxy_user_config=mapr=* \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-catalog_service_host=${CATALOG_SERVICE_HOST} \
-be_port=${IMPALA_BACKEND_PORT} \
-mem_limit=100% \
-disable_admission_control=true \
-kerberos_reinit_interval=60 \
-principal=mapr/master.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

# These impact the state store daemon and can be optionally changed
IMPALA_STATE_STORE_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-catalog_service_host=${CATALOG_SERVICE_HOST} \
-kerberos_reinit_interval=60 \
-principal=mapr/master.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

IMPALA_CATALOG_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-kerberos_reinit_interval=60 \
-principal=mapr/master.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

node1.localdomain

 

IMPALA_SERVER_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore \
-authorized_proxy_user_config=mapr=* \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-catalog_service_host=${CATALOG_SERVICE_HOST} \
-be_port=${IMPALA_BACKEND_PORT} \
-mem_limit=100% \
-disable_admission_control=true \
-kerberos_reinit_interval=60 \
-principal=mapr/node1.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

# These impact the state store daemon and can be optionally changed
IMPALA_STATE_STORE_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-catalog_service_host=${CATALOG_SERVICE_HOST} \
-kerberos_reinit_interval=60 \
-principal=mapr/master.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

IMPALA_CATALOG_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-kerberos_reinit_interval=60 \
-principal=mapr/master.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

node2.localdomain

 

IMPALA_SERVER_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore \
-authorized_proxy_user_config=mapr=* \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-catalog_service_host=${CATALOG_SERVICE_HOST} \
-be_port=${IMPALA_BACKEND_PORT} \
-mem_limit=100% \
-disable_admission_control=true \
-kerberos_reinit_interval=60 \
-principal=mapr/node2.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

# These impact the state store daemon and can be optionally changed
IMPALA_STATE_STORE_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-catalog_service_host=${CATALOG_SERVICE_HOST} \
-kerberos_reinit_interval=60 \
-principal=mapr/master.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

IMPALA_CATALOG_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-kerberos_reinit_interval=60 \
-principal=mapr/master.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab


"

 

node3.localdomain

 

IMPALA_SERVER_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore \
-authorized_proxy_user_config=mapr=* \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-catalog_service_host=${CATALOG_SERVICE_HOST} \
-be_port=${IMPALA_BACKEND_PORT} \
-mem_limit=100% \
-disable_admission_control=true \
-kerberos_reinit_interval=60 \
-principal=mapr/node3.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

# These impact the state store daemon and can be optionally changed
IMPALA_STATE_STORE_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-catalog_service_host=${CATALOG_SERVICE_HOST} \
-kerberos_reinit_interval=60 \
-principal=mapr/master.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

IMPALA_CATALOG_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-kerberos_reinit_interval=60 \
-principal=mapr/master.localdomain@EXAMPLE.COM \
-keytab_file=/opt/mapr/conf/mapr.keytab
"

I followed the document - Enable Kerberos Authentication  

 

Can anyone please help me out here and let me know what I have done wrong or missed !!. Thanks !!

Outcomes