AnsweredAssumed Answered

Spark log4j configuration in oozie

Question asked by mvince on Sep 21, 2017
Latest reply on Sep 21, 2017 by vmeghraj


i`m trying to configure custom logging in my spark jobs via files.

I updated in ../spark-1.6.1/conf/

and now logging works if I run my spark jobs with spark-submit command


However, I if I try to run spark action in oozie workflows, my is not applied

I tried to copy/link from spark-1.6.1/conf to oozie-4.2.0/conf/spark-conf and to oozie-4.2.0/conf


I also tried to change spark-defaults config to add row




but I`m getting errors:

Unrecognized option '-Dlog4j.configuration=/opt/mapr/spark/spark-1.6.1/conf/'.



Unrecognized option '-Dlog4j.configurationFile=/opt/mapr/spark/spark-1.6.1/conf/'.

I`m using MapR community 5.1 with oozie 4.2.0 and spark 1.6.1