AnsweredAssumed Answered

Bug: Livy with multiple Spark versions

Question asked by dodoman on Dec 12, 2016
Latest reply on Dec 14, 2016 by dodoman

Hi there,

 

here a small bug report of an issue happend to us:

 

Subject:

Hue Livy has issues if you have multiple spark versions installed

 

Description:

Livy Server is not able to set SPARK_HOME if you have multiple versions of spark installed.

 

Command to start manually:

/opt/mapr/hue/hue-3.9.0/bin/hue.sh livy_server start

 

Error which the manually executed command show:

 

/opt/mapr/hue/hue-3.9.0/bin/hue.sh: line 38: [: /opt/mapr/spark/spark-1.5.2: binary operator expected

Cannot export spark home!

 

Affected Versions:

All MapR Versions >= 5 

What warden shows: (Nothing, it just shows you that livy is running fine ??? So this is also a bug in warden)

 

Where does this error come from?

/opt/mapr/hue/hue-3.9.0/bin/hue.sh have a bad line of code in selecting the location of SPARK_HOME.

 

Code:

#get spark_home

  if [ $command == "livy_server" ]; then

    if [ -z "$SPARK_HOME" ]; then

        if [ -d /opt/mapr/spark/spark-* ]; then

          cd /opt/mapr/spark/spark-*

          SPARK_HOME=`pwd`

          export SPARK_HOME=`pwd`

        else

          echo "Cannot export spark home!"

          exit 0

        fi

    fi

    export PATH="$PATH:$SPARK_HOME/bin"

  fi

 

Reason why it fails:

If statements require binary operators and are not working with lists

 

Reason why you get 2 spark versions:

This is because of an update of the OS and the mapr-repositories. 

Our Operating Team is running a patch each week. 

So what the repository and deployment team does not know is, that you are forcing a version upgrade for spark. Other components stay at the major version while spark is upgrading from 1.5 to 1.6 instead of updating the 1.5.DATE package to 1.5.NEWDATE Package.

 

 

Fix:

Create a spark version file like you do for the most applications like:

HBase or Drill

 

Workaround until the patch comes:

Login to the linux server which is running livy.

Switch to the mapr user.

vim ~/.bash_profile

 

add for spark 1.5.2:

 

export SPARK_HOME=/opt/mapr/spark/spark-1.5.2

 

Newer version add the version number:

export SPARK_HOME=/opt/mapr/spark/spark-VERSION

Outcomes