AnsweredAssumed Answered

Pyspark application issues wih yarn cluster mode

Question asked by srinivasp on Jan 24, 2018
Latest reply on Jan 26, 2018 by cathy

Hi , We are using MapR5.2 ,Spark1.6.1 .

PySpark applications are running in 'Yarn Client' mode , but facing issue in 'Yarn Cluster' mode

we are getting python syntax issues .

Traceback (most recent call last):
 dir/usercache/padals06/appcache/application_1516777332998_0047/container_e359_1516777332998_0047_02_000001/__pyfiles__/Utilities.py", line 133
    unprocessed_experimentMetadata_Dict = { eachFileS3RawObjectId: experimentMetadata_Dict[eachFileS3RawObjectId] for eachFileS3RawObjectId in unproccessed_s3_files_keys }
                                          ^
SyntaxError: invalid syntax
End of LogType:stdout .

 

We are passing below parameter , still facing the issue .

export PYSPARK_PYTHON=/usr/local/bin/python2.7
export PYSPARK_DRIVER_PYTHON=/usr/local/bin/python2.7

Outcomes