here a small bug report of an issue happend to us:
Hue Livy has issues if you have multiple spark versions installed
Livy Server is not able to set SPARK_HOME if you have multiple versions of spark installed.
Command to start manually:
/opt/mapr/hue/hue-3.9.0/bin/hue.sh livy_server start
Error which the manually executed command show:
/opt/mapr/hue/hue-3.9.0/bin/hue.sh: line 38: [: /opt/mapr/spark/spark-1.5.2: binary operator expected
Cannot export spark home!
All MapR Versions >= 5
What warden shows: (Nothing, it just shows you that livy is running fine ??? So this is also a bug in warden)
Where does this error come from?
/opt/mapr/hue/hue-3.9.0/bin/hue.sh have a bad line of code in selecting the location of SPARK_HOME.
if [ $command == "livy_server" ]; then
if [ -z "$SPARK_HOME" ]; then
if [ -d /opt/mapr/spark/spark-* ]; then
echo "Cannot export spark home!"
Reason why it fails:
If statements require binary operators and are not working with lists
Reason why you get 2 spark versions:
This is because of an update of the OS and the mapr-repositories.
Our Operating Team is running a patch each week.
So what the repository and deployment team does not know is, that you are forcing a version upgrade for spark. Other components stay at the major version while spark is upgrading from 1.5 to 1.6 instead of updating the 1.5.DATE package to 1.5.NEWDATE Package.
Create a spark version file like you do for the most applications like:
HBase or Drill
Workaround until the patch comes:
Login to the linux server which is running livy.
Switch to the mapr user.
add for spark 1.5.2:
Newer version add the version number: