i`m trying to configure custom logging in my spark jobs via log4j.properties files.
I updated log4j.properties in ../spark-1.6.1/conf/log4j.properties
and now logging works if I run my spark jobs with spark-submit command
However, I if I try to run spark action in oozie workflows, my log4j.properties is not applied
I tried to copy/link log4j.properties from spark-1.6.1/conf to oozie-4.2.0/conf/spark-conf and to oozie-4.2.0/conf
I also tried to change spark-defaults config to add row
but I`m getting errors:
Unrecognized option '-Dlog4j.configuration=/opt/mapr/spark/spark-1.6.1/conf/log4j.properties'.
Unrecognized option '-Dlog4j.configurationFile=/opt/mapr/spark/spark-1.6.1/conf/log4j.properties'.
I`m using MapR community 5.1 with oozie 4.2.0 and spark 1.6.1