AnsweredAssumed Answered

Bug: Livy with multiple Spark versions

Question asked by dodoman on Dec 12, 2016
Latest reply on Dec 14, 2016 by dodoman

Hi there,


here a small bug report of an issue happend to us:



Hue Livy has issues if you have multiple spark versions installed



Livy Server is not able to set SPARK_HOME if you have multiple versions of spark installed.


Command to start manually:

/opt/mapr/hue/hue-3.9.0/bin/ livy_server start


Error which the manually executed command show:


/opt/mapr/hue/hue-3.9.0/bin/ line 38: [: /opt/mapr/spark/spark-1.5.2: binary operator expected

Cannot export spark home!


Affected Versions:

All MapR Versions >= 5 

What warden shows: (Nothing, it just shows you that livy is running fine ??? So this is also a bug in warden)


Where does this error come from?

/opt/mapr/hue/hue-3.9.0/bin/ have a bad line of code in selecting the location of SPARK_HOME.



#get spark_home

  if [ $command == "livy_server" ]; then

    if [ -z "$SPARK_HOME" ]; then

        if [ -d /opt/mapr/spark/spark-* ]; then

          cd /opt/mapr/spark/spark-*


          export SPARK_HOME=`pwd`


          echo "Cannot export spark home!"

          exit 0



    export PATH="$PATH:$SPARK_HOME/bin"



Reason why it fails:

If statements require binary operators and are not working with lists


Reason why you get 2 spark versions:

This is because of an update of the OS and the mapr-repositories. 

Our Operating Team is running a patch each week. 

So what the repository and deployment team does not know is, that you are forcing a version upgrade for spark. Other components stay at the major version while spark is upgrading from 1.5 to 1.6 instead of updating the 1.5.DATE package to 1.5.NEWDATE Package.




Create a spark version file like you do for the most applications like:

HBase or Drill


Workaround until the patch comes:

Login to the linux server which is running livy.

Switch to the mapr user.

vim ~/.bash_profile


add for spark 1.5.2:


export SPARK_HOME=/opt/mapr/spark/spark-1.5.2


Newer version add the version number:

export SPARK_HOME=/opt/mapr/spark/spark-VERSION