AnsweredAssumed Answered

Spark Structured Streaming + Stream-Stream Joins

Question asked by PETER.EDIKE on Jun 19, 2018
Latest reply on Jun 19, 2018 by vmeghraj

Hello Everyone,

 

I am trying to implement a Stream-Stream join Use Case on a MapR Cluster 6.0.1 MEP 5.0 and I keep getting the following exception

 

warning: there was one deprecation warning; re-run with -deprecation for details
org.apache.spark.sql.AnalysisException: Inner join between two streaming DataFrames/Datasets is not supported;;
Join Inner, (TransactionId#358 = Id#51)
:- EventTimeWatermark date_entered_trans#180: timestamp, interval 10 minutes
: +- Project [Id#51, BankId#52, PaymentRefNum#53, BankCode#54, BankCBNCode#55, BankName#56, TerminalOwnerCode#57, TerminalOwnerName#58, CurrencyCode#59, CurrencyName#60, PaymentDate#61, ResponseCode#62, TransactionAmount#63, ApprovedAmount#64, Surcharge#65, SurchargeCurrencyCode#66, TransactionType#67, TerminalId#68, RetrievalReferenceNumber#69, EncryptedPAN#70, HashedPAN#71, MaskedPAN#72, CustomerName#73, CustomerEmail#74, ... 41 more fields]

It says inner join between two streaming dataframes/datasets is not supported. Please I did like to know for sure if this feature is not supported in the latest version of MapR.

Outcomes