AnsweredAssumed Answered

How to use log retention policy while developing streaming application in spark

Question asked by avinashyeluri on Oct 18, 2017
Latest reply on Oct 19, 2017 by john.humphreys

Hi ,have question on streaming application with spark/mapr-streams.


stderr :  Total file length is 3031893 bytes.

stdout : Total file length is 181919 bytes.


i have specified below configurations while running application , but stderr log file was not rolling .. do i have to configure with log4j properties to sync up ?  can any one guide me how to make the log file not to increase

 --conf "spark.executor.logs.rolling.maxRetainedFiles=4" \  --conf "spark.executor.logs.rolling.strategy=time" \  --conf "spark.executor.logs.rolling.time.interval=hourly" \