Velumani

YARN log aggregation

Discussion created by Velumani on Jun 20, 2016
Latest reply on Jul 19, 2016 by asukhenko

Hi,

     Can some one explain what is YARN log aggregation and how it works? In my mapr sandbox how can I find whether log aggregation is enabled or not? "yarn.log-aggregation-enable"  property is not set in /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/yarn-site.xml file. As per this document (YARN Log Aggregation - MapR 5.0 Documentation - doc.mapr.com ) that is not enabled by default but I was able to see the logs in YARN HistoryServer  UI without that property set, but when I added the property in yarn-site.xml, the spark job was not completing when I submit using yarn-client or yarn-cluster and I am not able to see those logs in UI . Previously it used to work.

 

Below property I added in yarn-site.xml

<property>

   <name>yarn.log-aggregation-enable</name>

   <value>true</value>

  </property>

Outcomes