AnsweredAssumed Answered

/opt/mapr/lib/spring jars conflict with my own job with Mapr5.1

Question asked by perttvil on Jun 13, 2016
Latest reply on Feb 22, 2018 by maprcommunity

I just installed mapr 5.1 cluster, and I noticed that /opt/mapr/lib has spring 3 jars. Mapr5.0 did not have these jars.

 

Spring 3 does not work with java 8 (problems with asm). And I am using spring 4 with java 8 in my spark job. My spark job adds spring 4 jars to the job classpath and everything worked well with mapr 5.0.

 

It seems /opt/mapr/lib/spring*3* jars are added to spark job classpath automatically and those jars are in the start of classpath, so even though my spark job includes spring 4 jars, spark 3 jars are used from /opt/mapr/lib => Job does not work.

 

This might have something to do with LD_LIBRARY_PATH environment variable set in hadoop_config.sh and other config

scripts under hadoop directory.

 

There are no references to this directory under /opt/mapr/spark/spark-1.6.0/ directory so mapr distribution must add then to classpath somewhere.

 

Is there a safe way to remove spring 3 jars from spark job classpath?

Outcomes