Architecting Serverless Big Data Solutions Using Google Dataflow
|
Dataflow represents a fundamentally different approach to Big Data processing than computing engines such as Spark. Dataflow is serverless and fully-managed, meaning that provisioning resources and scaling can be transparent to the data architect.
Dataflow allows developers to process and transform data using easy, intuitive APIs. Dataflow is built on the Apache Beam architecture and unifies batch as well as stream processing of data. In this course, Architecting Serverless Big Data Solutions Using Google Dataflow, you will be exposed to the full potential of Cloud Dataflow and its radically innovative programming model. You will start this course off with a basic understanding of how Dataflow works for serverless compute. You’ll study the Apache Beam API used to build pipelines and understand what data sources, sinks, and transformations are. You’ll study the stages in a Dataflow pipeline and visualize it as a directed-acyclic graph. Next, you’ll use Apache Beam APIs to build pipelines for data transformations in both Java as well as Python and execute these pipelines locally and on the cloud. You’ll integrate your pipelines with other GCP services such as BigQuery and see how you can monitor and debug slow pipeline stages. Additionally, you’ll study different pipeline architectures such as branching and pipelines using side inputs. You’ll also see how you can apply windowing operations to perform aggregations on our data. Finally, you’ll work with Dataflow without writing any code using pre-built Dataflow templates that Google offers for common operations. At the end of this course, you should be comfortable using Dataflow pipelines to transform and process your data and integrate your pipelines with other Google services.
Author Name: Janani Ravi
Author Description:
Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework. After spending years working in tech in the Bay Area, New York, and Singapore at companies such as Microsoft, Google, and Flipkart, Janani finally decided to combine her love for technology with her passion for teaching. She is now the co-founder of Loonycorn, a content studio focused on providing … more
User Reviews
Be the first to review “Architecting Serverless Big Data Solutions Using Google Dataflow”
Related Products
IBM: Apache Spark for Data Engineering and Machine Learning
This short course introduces you to the fundamentals of Data Engineering and Machine Learning with Apache Spark, including Spark Structured Streaming, ETL for Machine Learning (ML) Pipelines, and Spark ML. By the end of the course, you will have hands-on experience applying Spark skills to ETL and ML workflows.
AdelaideX: Big Data Analytics
Learn key technologies and techniques, including R and Apache Spark, to analyse large-scale data sets to uncover valuable business information.
AdelaideX: Big Data Fundamentals
Learn how big data is driving organisational change and essential analytical tools and techniques, including data mining and PageRank algorithms.
Building Scalable Data Pipelines with Kafka
This course brings any Big Data newbie up to speed on one of the most widely used open-source platform to build data pipelines - Apache Kafka.
AdelaideX: Computational Thinking and Big Data
Learn the core concepts of computational thinking and how to collect, clean and consolidate large-scale datasets.
HKUSTx: Foundations of Data Analytics
Learn the fundamental techniques for data analytics and to be prepared for learning and applying more advanced big data technologies.
There are no reviews yet.