Stream processing applications work with continuously updated data and react to changes in real-time. Data frames in Spark 2.x support infinite data, thus effectively unifying batch and streaming applications. In this course, Structured Streaming in Apache Spark 2, you’ll focus on using the tabular data frame API to work with streaming, unbounded datasets using the same APIs that work with bounded batch data. First, you’ll start off by understanding how structured streaming works and what makes it different and more powerful than traditional streaming applications; the basic streaming architecture and the improvements included in structured streaming allowing it to react to data in real-time. Then you’ll create triggers to evaluate streaming results and output modes to write results out to file or screen. Next, you’ll discover how you can build streaming pipelines using Spark by studying event time aggregations, grouping and windowing functions, and how to perform join operations between batch and streaming data. You’ll even work with real Twitter streams and perform analysis on trending hashtags on Twitter. Finally, you’ll then see how Spark stream processing integrates with the Kafka distributed publisher-subscriber system by ingesting Twitter data from a Kafka producer and process it using Spark Streaming. By the end of this course, you’ll be comfortable performing analysis of stream data using Spark’s distributed analytics engine and its high-level structured streaming API.
Author Name: Janani Ravi
Author Description:
Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework. After spending years working in tech in the Bay Area, New York, and Singapore at companies such as Microsoft, Google, and Flipkart, Janani finally decided to combine her love for technology with her passion for teaching. She is now the co-founder of Loonycorn, a content studio focused on providing … more

Table of Contents

• Course Overview
  2mins
• Understanding the High Level Streaming API in Spark 2.x
  47mins
• Building Advanced Streaming Pipelines Using Structured Streaming
  59mins
• Integrating Apache Kafka with Structured Streaming
  22mins