MapR Best Practices Enquiry Storing Historical Data

Discussion created by Lilia on Jan 11, 2017
Latest reply on Jan 27, 2017 by MichaelSegel

Business Case: we need to store data in physical storage (on disk) emitted in MapR Stream so we have in place the historical data captured for analytical purposes. MapR Stream parameter ttl provides possibility to store data for extended from default upto unlimited time.

Concerns: Using TTL for MapR streams persists data in MapR FS for time defined, after expiration data is lost and removed from disk. Also on MapR Stream level there is not possibility to handle PCI and PII data encryption explicitly. Solution: We are looking into the possibility of creating ‘staging area’ on MapR FS with log table objects storing raw data for analytic purposes.

Best Practices: Please advise us on the best practices and options log tables in staging area vs MapR stream ttl fulfilling the business case.