AnsweredAssumed Answered

Spark log4j configuration in oozie

Question asked by mvince on Sep 21, 2017
Latest reply on Sep 21, 2017 by vmeghraj

Hi

i`m trying to configure custom logging in my spark jobs via log4j.properties files.

I updated log4j.properties in ../spark-1.6.1/conf/log4j.properties

and now logging works if I run my spark jobs with spark-submit command

 

However, I if I try to run spark action in oozie workflows, my log4j.properties is not applied

I tried to copy/link log4j.properties from spark-1.6.1/conf to oozie-4.2.0/conf/spark-conf and to oozie-4.2.0/conf

 

I also tried to change spark-defaults config to add row

spark.driver.extraJavaOptions  

-Dlog4j.configuration=/opt/mapr/spark/spark-1.6.1/conf/log4j.property

and
-Dlog4j.configurationFile=/opt/mapr/spark/spark-1.6.1/conf/log4j.property

but I`m getting errors:

Unrecognized option '-Dlog4j.configuration=/opt/mapr/spark/spark-1.6.1/conf/log4j.properties'.

 

 

Unrecognized option '-Dlog4j.configurationFile=/opt/mapr/spark/spark-1.6.1/conf/log4j.properties'.



I`m using MapR community 5.1 with oozie 4.2.0 and spark 1.6.1

Outcomes