AnsweredAssumed Answered

Spark Iterating RDD over another RDD with filter conditions .scala

Question asked by madhureddy915 on Aug 3, 2015
Latest reply on Aug 10, 2015 by dgomerman

I wants to iterate one BIG RDD with small RDD with some additional filter conditions . the below code is working fine but the process is running only with Driver and Not spread-ed across the nodes . So please suggest any other approach ?

    val cross = titlesRDD.cartesian(brRDD).cache()
     val matching = cross.filter{ case( x, br) =>
        ((br._1 == "0") &&
       (((br._2 ==((x._4))) &&
        ((br._3 exists (x._5)) || ((br._3).head==""))