SLIDES: Architecting a hybrid cloud application using a global publish-subscribe streaming message system

File uploaded by slimbaltagi on Feb 3, 2018
Version 1Show Document
  • View in full screen mode

Presentation by Mathieu Dumoulin at the Strata+Hadoop World on December 7th 2016 in Singapore. 

"Hybrid cloud architectures offer the opportunity to build applications and workflows that marry the security and control of an on-premises platform with the flexibility to move workloads to a public cloud to scale capacity as needed. Hybrid cloud implementations are useful in a variety of scenarios, including scaling existing applications to meet growing demands, building a disaster recovery solution, powering cloud-native and mobile applications that need to scale with traffic spikes while maintaining security and compliance requirements, and facilitating Agile development practices.

Underlying all hybrid cloud use cases is the need for an efficient and scalable stream message delivery system to connect the on-premises data center with the public cloud deployment(s). Kafka and other publish-subscribe message systems offer a natural paradigm for integrating on-premises and on-cloud workloads. Moreover, the integration of stream processing frameworks, such as Spark Streaming and Flink, with these Kafka-based message systems enables real-time analytics to be performed at scale in hybrid cloud environments.

Mathieu Dumoulin describes how a publish-subscribe message system can be used to architect a hybrid cloud solution, how a streaming message system can be used in a complex hybrid cloud application, and how to combine sensor data from global IoT applications running in multiple public clouds. Along the way, Mathieu also covers how to leverage on-premises analysis in a stream processing framework on Hadoop, such as Spark Streaming or Flink, and the trade-offs between some the competing message systems and stream processing frameworks."