AnsweredAssumed Answered

[Saprk 1.4.1 on MapR 5] native library configuration is missing

Question asked by dafox on Sep 23, 2015
Latest reply on Mar 6, 2016 by dafox
Hello,

I have installeld Spark 1.4.1 on MapR 5. When I run Spark application the following warning is displayed:

    WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

The problem can be in **spark-env.sh** script, it is not setting properly native-hadoop libararies to enviroment variables. Variable is exported before it's content is initialized.

    export SPARK_LIBRARY_PATH=$MAPR_HADOOP_JNI_PATH
    # few lines later
    MAPR_HADOOP_JNI_PATH=`hadoop jnipath`

Actually it looks like this variable is not used by Spark any more and `LD_LIBRARY_PATH` varible is needed :

    export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$MAPR_HADOOP_JNI_PATH"

Do I missing or misunderstanding something (e.g. `spark-env.sh` purpose)?

Thanks!

Outcomes