AnsweredAssumed Answered

Spark - Scala code works perfectly in Eclipse, but throwing error when running in MapR cluster.

Question asked by sijabalakrishnan on Jul 22, 2017
Latest reply on Jul 25, 2017 by maprcommunity

I am running a code in Eclipse which involves data extraction from a URL and saving into a text file ( File creation).

And surprisingly its throwing a "file not found" error for Output file creation in MapR cluster. This program runs perfectly in eclipse. 

ERROR MESSAGE

Exception in thread "Thread-52" Exception in thread "Thread-51" java.io.FileNotFoundException: /hdfs1/output/put/99694d2ea217.docx (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at scala.sys.process.ProcessBuilderImpl$FileOutput$$anonfun$$lessinit$greater$3.apply(ProcessBuilderImpl.scala:33)
at scala.sys.process.ProcessBuilderImpl$FileOutput$$anonfun$$lessinit$greater$3.apply(ProcessBuilderImpl.scala:33)
at scala.sys.process.ProcessBuilderImpl$OStreamBuilder$$anonfun$$lessinit$greater$4.apply(ProcessBuilderImpl.scala:38)
at scala.sys.process.ProcessBuilderImpl$OStreamBuilder$$anonfun$$lessinit$greater$4.apply(ProcessBuilderImpl.scala:38)
at scala.sys.process.ProcessBuilderImpl$ThreadBuilder$$anonfun$1.apply$mcV$sp(ProcessBuilderImpl.scala:58)
at scala.sys.process.ProcessImpl$Spawn$$anon$1.run(ProcessImpl.scala:23)

 

Resolution method tried.

    1. I have checked all the permissions for the folder
    2. Tried changing path
    3. Also, tried giving location both implicitly and explicitly within the code.

Outcomes