How to limit the Hive log size using RFA instead of default DRFA

Document created by Hao Zhu Employee on Feb 7, 2016
Version 1Show Document
  • View in full screen mode

Author: Hao Zhu

 

Original Publication Date: December 3, 2015

 

Environment :

Hive 1.2 on MapR

Goal:

By default, MapR Hive is using DRFA(Daily Rolling File Appender) for log4j. The template for DRFA settings are in /opt/mapr/hive/hive-<version>/conf/hive-log4j.properties.template

Administrators can copy hive-log4j.properties.template to hive-log4j.properties in "conf" directory and make the changes as they want.

 

However if the daily Hive log is too large and may potentially fill up all the disk space, we can use RFA(Rolling File Appender) instead to set a max size of each log and also the total number of logs.

Solution:

1. Copy /opt/mapr/hive/hive-<version>/conf/hive-log4j.properties.template to hive-log4j.properties in the same directory.

 

2.  Change the DRFA log4j settings to RFA ones.

For example, if we want to limit the size of each hive log to 1MB, and number of hive logs to 3, below is a complete example:

 

[root@v1 conf]# cat hive-log4j.properties

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements. See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership. The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License. You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

 

# Define some default values that can be overridden by system properties

hive.log.threshold=ALL

hive.root.logger=INFO,RFA

hive.log.dir=/opt/mapr/hive/hive-1.2/logs/${user.name}

hive.log.file=hive.log

 

# Define the root logger to the system property "hadoop.root.logger".

log4j.rootLogger=${hive.root.logger}, EventCounter

 

# Logging Threshold

log4j.threshold=${hive.log.threshold}

 

#

# Rolling File Appender - cap space usage at 1MB

#

log4j.appender.RFA=org.apache.log4j.RollingFileAppender

log4j.appender.RFA.File=${hive.log.dir}/${hive.log.file}

log4j.appender.RFA.MaxFileSize=1MB
log4j.appender.RFA.MaxBackupIndex=3

log4j.appender.RFA.layout=org.apache.log4j.PatternLayout

 

# Pattern format: Date LogLevel LoggerName LogMessage

# Debugging Pattern format

log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n

 

 

#

# console

# Add "console" to rootlogger above if you want to use this

#

 

log4j.appender.console=org.apache.log4j.ConsoleAppender

log4j.appender.console.target=System.err

log4j.appender.console.layout=org.apache.log4j.PatternLayout

log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} [%t]: %p %c{2}: %m%n

log4j.appender.console.encoding=UTF-8

 

#custom logging levels

#log4j.logger.xxx=DEBUG

 

#

# Event Counter Appender

# Sends counts of logging messages at different severity levels to Hadoop Metrics.

#

log4j.appender.EventCounter=org.apache.hadoop.hive.shims.HiveEventCounter

 

 

log4j.category.DataNucleus=ERROR,RFA

log4j.category.Datastore=ERROR,RFA

log4j.category.Datastore.Schema=ERROR,RFA

log4j.category.JPOX.Datastore=ERROR,RFA

log4j.category.JPOX.Plugin=ERROR,RFA

log4j.category.JPOX.MetaData=ERROR,RFA

log4j.category.JPOX.Query=ERROR,RFA

log4j.category.JPOX.General=ERROR,RFA

log4j.category.JPOX.Enhancer=ERROR,RFA

 

 

# Silence useless ZK logs

log4j.logger.org.apache.zookeeper.server.NIOServerCnxn=WARN,RFA

log4j.logger.org.apache.zookeeper.ClientCnxnSocketNIO=WARN,RFA

1 person found this helpful

Attachments

    Outcomes