AnsweredAssumed Answered

Spark Shell not starting : Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream

Question asked by sijabalakrishnan on Aug 1, 2017
Latest reply on Mar 1, 2018 by shrilesh.naik
Branched to a new discussion

Hey guys, 

I

My issue is that my spark stopped working only a couple of days back. This was starting up correctly before.

I went through the "spark-env.sh", but everything seems to be in place.

 

I am using MapR - 5.2 cluster with spark - 2.0.1

I have pasted the screenshot of the error message below :

 

=> details of env file : $SPARK_HOME/conf/spark-env.sh

 

# Set the spark attributes
export SPARK_HOME=/opt/mapr/spark/spark-2.0.1

 

# Load the hadoop version attributes
source /opt/mapr/spark/spark-2.0.1/mapr-util/hadoop-version-picker.sh
export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.7.0

 

export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
export SPARK_CONF_DIR=/home/mapr
# Enable mapr impersonation
export MAPR_IMPERSONATION_ENABLED=1

 

MAPR_HADOOP_CLASSPATH=`hadoop classpath`:/opt/mapr/lib/slf4j-log4j12-1.7.5.jar:

 

MAPR_HADOOP_JNI_PATH=`hadoop jnipath`
MAPR_SPARK_CLASSPATH="$MAPR_HADOOP_CLASSPATH"

 

SPARK_MAPR_HOME=/opt/mapr

 

export SPARK_LIBRARY_PATH=$MAPR_HADOOP_JNI_PATH
export LD_LIBRARY_PATH="$MAPR_HADOOP_JNI_PATH:$LD_LIBRARY_PATH"

 

# Load the classpath generator script
source /opt/mapr/spark/spark-2.0.1/mapr-util/generate-classpath.sh

 

# Calculate hive jars to include in classpath
generate_compatible_classpath "spark" "2.0.1" "hive"
MAPR_HIVE_CLASSPATH=${generated_classpath}
if [ ! -z "$MAPR_HIVE_CLASSPATH" ]; then
  MAPR_SPARK_CLASSPATH="$MAPR_SPARK_CLASSPATH:$MAPR_HIVE_CLASSPATH"
fi

 

# Calculate hbase jars to include in classpath
generate_compatible_classpath "spark" "2.0.1" "hbase"
MAPR_HBASE_CLASSPATH=${generated_classpath}
if [ ! -z "$MAPR_HBASE_CLASSPATH" ]; then
  MAPR_SPARK_CLASSPATH="$MAPR_SPARK_CLASSPATH:$MAPR_HBASE_CLASSPATH"
  SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Dspark.executor.extraClassPath=$MAPR_HBASE_CLASSPATH -Dspark.driver.extraClassPath=$MAPR_HBASE_CLASSPATH"
fi

 

# Set SPARK_DIST_CLASSPATH
#export SPARK_DIST_CLASSPATH=$MAPR_SPARK_CLASSPATH
export SPARK_DIST_CLASSPATH=`hadoop classpath`
# Security status
source /opt/mapr/conf/env.sh
if [ "$MAPR_SECURITY_STATUS" = "true" ]; then
  SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Dhadoop.login=hybrid -Dmapr_sec_enabled=true"
fi

 

# scala
export SCALA_VERSION=2.11
export SPARK_SCALA_VERSION=$SCALA_VERSION
export SCALA_HOME=/opt/mapr/spark/spark-2.0.1/scala
export SCALA_LIBRARY_PATH=$SCALA_HOME/lib

 

# Use a fixed identifier for pid files
export SPARK_IDENT_STRING="mapr"

 

#########################################################################################################
#    :::CAUTION::: DO NOT EDIT ANYTHING ON OR ABOVE THIS LINE
#########################################################################################################

 


#
# MASTER HA SETTINGS
#
#export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER  -Dspark.deploy.zookeeper.url=<zookeerper1:5181,zookeeper2:5181,..> -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.sasl.client=false"

 


# MEMORY SETTINGS
export SPARK_DAEMON_MEMORY=1g
export SPARK_WORKER_MEMORY=16g

 

# Worker Directory
export SPARK_WORKER_DIR=$SPARK_HOME/tmp

 

# Environment variable for printing spark command everytime you run spark.Set to "1" to print.
# export SPARK_PRINT_LAUNCH_COMMAND=1

Outcomes