Architecting Big Data Solutions Using Google Dataproc
|
Dataproc is Google’s managed Hadoop offering on the cloud. This course teaches you how the separation of storage and compute allows you to utilize clusters more efficiently purely for processing data and not for storage.
When organizations plan their move to the Google Cloud Platform, Dataproc offers the same features but with additional powerful paradigms such as separation of compute and storage. Dataproc allows you to lift-and-shift your Hadoop processing jobs to the cloud and store your data separately on Cloud Storage buckets, thus effectively eliminating the requirement to keep your clusters always running. In this course, Architecting Big Data Solutions Using Google Dataproc, you’ll learn to work with managed Hadoop on the Google Cloud and the best practices to follow for migrating your on-premise jobs to Dataproc clusters. First, you’ll delve into creating a Dataproc cluster and configuring firewall rules to enable you to access the cluster manager UI from your local machine. Next, you’ll discover how to use the Spark distributed analytics engine on your Dataproc cluster. Then, you’ll explore how to write code in order to integrate your Spark jobs with BigQuery and Cloud Storage buckets using connectors. Finally, you’ll learn how to use your Dataproc cluster to perform extract, transform, and load operations using Pig as a scripting language and work with Hive tables. By the end of this course, you’ll have the necessary knowledge to work with Google’s managed Hadoop offering and have a sound idea of how to migrate jobs and data on your on-premise Hadoop cluster to the Google Cloud.
Author Name: Janani Ravi
Author Description:
Janani has a Masters degree from Stanford and worked for 7+ years at Google. She was one of the original engineers on Google Docs and holds 4 patents for its real-time collaborative editing framework. After spending years working in tech in the Bay Area, New York, and Singapore at companies such as Microsoft, Google, and Flipkart, Janani finally decided to combine her love for technology with her passion for teaching. She is now the co-founder of Loonycorn, a content studio focused on providing … more
Table of Contents
- Course Overview
2mins - Introducing Google Dataproc for Big Data on the Cloud
42mins - Running Hadoop MapReduce Jobs on Google Dataproc
49mins - Working with Apache Spark on Google Dataproc
24mins - Working with Pig and Hive on Google Dataproc
18mins
User Reviews
Be the first to review “Architecting Big Data Solutions Using Google Dataproc”
Related Products
IBM: Apache Spark for Data Engineering and Machine Learning
This short course introduces you to the fundamentals of Data Engineering and Machine Learning with Apache Spark, including Spark Structured Streaming, ETL for Machine Learning (ML) Pipelines, and Spark ML. By the end of the course, you will have hands-on experience applying Spark skills to ETL and ML workflows.
Data Science in Production: Building Scalable Model Pipelines
Learn how to build scalable model pipelines for data science in production environments, focusing on best practices for deployment and management.
AdelaideX: Computational Thinking and Big Data
Learn the core concepts of computational thinking and how to collect, clean and consolidate large-scale datasets.
Big Data Systems
Learn about the architecture and management of big data systems.
CSE 8803 Special Topics: Big Data
Explore advanced topics in big data technologies, architectures, and analytical methods.
The No Nonsense Introduction To Big Data, Hadoop and Streaming
A straightforward introduction to big data, Hadoop, and streaming technologies, focusing on fundamental concepts and practical applications.
There are no reviews yet.