Intro to Big Data AppHub: Kafka to Database & Database to HDFS App Templates - 2/8/17 - Remote Attendees welcome!

Document created by Patrick Moran on Feb 1, 2017
Version 1Show Document
  • View in full screen mode

SUMMARY

Date:Wednesday, February 8, 2017
Location:Webcast
Time:10:00am - 12:00pm PST
Registration Link:https://www.meetup.com/Big-Data-Hadoop-Ingest-Transform-Apex-Bay-Area-Online/events/237348666/

                        

ABOUT THE EVENT

Abstract: 

To make critical business decisions in real time, many businesses today rely on a variety of data, which arrives in large volumes. Variety and volume together make big data applications complex operations. Big data applications require businesses to combine transactional data with structured, semi-structured, and unstructured data for deep and holistic insights.

 

And, time is of the essence: to derive the most valuable insights and drive key decisions, large amounts of data have to be continuously ingested into Hadoop data lakes as well as other destinations. As a result, data ingestion poses the first challenge for businesses, which must be overcome before embarking on data analysis.

 

With its various Application Templates for ingestion, DataTorrent allows users to:

Ingest vast amounts of data with enterprise-grade operability and performance guarantees provided by its underlying Apache Apex framework. Those guarantees include fault tolerance, linear scalability, high throughput, low latency, and end-to-end exactly-once processing.

Quickly launch template applications to ingest raw data, while also providing an easy and iterative way to add business logic and such processing logic as parse, dedupe, filter, transform, enrich, and more to ingestion pipelines. Visualize various metrics on throughput, latency and app data in real-time throughout execution.

 

Template descriptions:
Kafka to Database Sync: The Kafka to Database Sync Application ingests string messages separated by '|' from configured Kafka topic and writes each message as a record in PostgreSQL DataBase.

 

Database to HDFS: The Database to HDFS Application Template continuously polls records from configured PostgreSQL table and writes records as comma separated lines in HDFS file(s).

 

Presenters:
Ashwin Putta, Product Manager at DataTorrent and Committer for Apache Apex

Sanjay Pujare, Engineer at DataTorrent

Dr. Munagala V. Ramanath ("Ram"), Committer for Apache Apex, Software Engineer at DataTorrent.

 

Agenda:

10:00 AM – 10:30 AM – Introduction to AppHub, AppHub principles, TCO, time to market - Ashwin

10:30 AM – 11:00 AM – Kafka to Database Sync - Ram

11:00 AM – 11:30 AM – Database to HDFS - Sanjay

11:30 AM – 12:00 PM – Upcoming templates, cloud, roadmap - Ashwin

 

LOGISTICAL DETAILS

Please RSVP for the event via meetup.com and follow the instructions in the event description.

 

MORE DETAILS

For deeper engagement with Apache Apexdownload, view past meetup webinarsslides, and docs

To reduce time to market, look at operable app-templates that you can quickly import and launch. 

Examples: HDFS-SyncKafka-HDFSHDFS-Line-CopyS3-HDFS and HDFS-Kafka.

Free DataTorrent Enterprise Edition for qualifying startups. Check it out!

Brought to you by DataTorrent, creators of Apache Apex.

Attachments

    Outcomes