Currently we have application where we get ~1500K message/day from upstream servers and then we perform some basic checks & enrichment from static data and store the messages in cache cluster as well as in RDBMS. We are using cache here to reduce the latency of requested data from downstreem systems and also to join the data.
We are thinking to get rid of this cache as it's getting bigger and very difficult to manage. Also data volume is increasing as well. We just doing some pocs with Kafka, it worked well for processing validation, enrichment etc but we also need on demand/ request based different data set joining like emp with Dept etc on different fields and craiteria. Kafka is not designed for request based data set join rather continues stream join. Please suggest how Mapr can help here ? What we need
1. Data join capabilities with ordering and pagination
2. Quick response time with in millis