I'm using MapR 5.2.0 in secure mode + MEP 3.0.0 (spark 2.1.0), in Spark UI, Storage tab is all blank and for all executors, storage property is always 0.
is there such know bug?
Just want to verify that you are looking at the Spark Web UI while there are active Spark jobs running. By default, information is only available for the duration of a job.
Yes my job is running, for instance I run a spark-shell this way
[myser@myhost ~]$ spark-shell --master yarn --files keystore.jks,truststore.jksimport org.apache.spark.storage.StorageLevel._
val the_rdd = sc.parallelize(1 to 10000000)the_rdd.setName("Foo")the_rdd.collect()
the_rdd.persist(DISK_ONLY)val l = the_rdd.filter( _ % 2==0)l.saveAsTextFile("export.txt")
and at the same time I go to "incomplete jobs" and in the job I can't have any output in storage tab, neither I get storage value in executors.
I have seen this behavior before. Persisting RDD in DISK is not shown in Spark Web UI. However, for other persisting levels it gets displayed (eg: MEMORY_ONLY). Also if you convert the RDD into Dataframe and try the same, then it works fine. Shall check this once again in the latest MapR Spark version and see if the behavior is same.
Hi Harikrishnan Cheneperth Kunhumveettil
I've tested also with standard StorageLevel so MEMORY_ONLY before and it does not either shows up in the Storage Tab.
Upgrading MapR is not an option for us now as we encounter this bug on a customer facing platform.
converting rdd to dataframe doesn't solve the issue, either.
Thanks for your answer.
Retrieving data ...