Rachel Silver

Spark 2.0 Released for Developer Preview

Blog Post created by Rachel Silver Employee on Jun 16, 2016

Release Announcement: Spark 2.0 Developer Preview

Date: June 16, 2016

 

Apache Spark 2.0 is now available on MapR! This is a Developer Preview release and not recommended for  production deployments.

 

What is a Developer Preview?

The MapR Developer Preview gives you the opportunity to work with new features and technologies that are still under active development. We are excited about the prospects of these technologies and look forward to working with users, customers, and the wider open source and partner community to enhance them.

 

Projects featured on these pages are not supported by MapR. We consider these upcoming projects to have some sharp edges and bugs, and to change significantly. Therefore, we do not recommend using these features in production systems at this point in time.

 

MapR will publish easy to install packages and basic installation/monitoring/managing instructions for technologies in Developer Preview. If you have any questions, please post them on the respective forums and/or on our Converge Community.

 

What New Features Has Spark 2.0 Introduced?

 

Structured APIs:

  • ex. DataFrame API:
    • Runs on the same engine as SparkSQL.
    • Allows access to data from a variety of different data sources.
    • Can run database-like operations or allow for passing in custom code.

 

Structured Streaming with Spark SQL Streaming:

  • Provides the ability to perform interactive queries against live streaming data.
  • Introduces the concept of repeated queries wherein a particular query is executed repeatedly against every incoming batch of DStreams (RDDs).
  • Output can now be aggregated in a stream for continuous applications.

 

Spark as a Compiler:

  • Whole-stage code generation provided by the second-generation Tungsten engine.
  • Eliminates the need for multiple JVM calls by flattening SQL queries into one single function evaluated as bytecode at runtime.

 

Documentation & Download

 

Internal Resources

 

Further Resources

 

Have a Question?

Ask in Answers or comment below.

 

Be the first to get notified about new releases

How to get notified of release, product and patch announcements

Outcomes