MapReduce and YARN
|
Master the concepts of MapReduce and YARN in Hadoop. Learn how they work together to process large datasets efficiently and manage cluster resources.
At a Glance
String together your understanding of Yet Another Resource Negotiator (YARN) by gaining exposure to MapReduce1, the tool-sets that start the processing of Big Data.
About This Course
Apache Hadoop is one of the most popular tools for big data processing. It has been successfully deployed in production by many companies for several years. Though Hadoop is considered a reliable, scalable, and cost-effective solution, it is constantly being improved by a large community of developers. As a result, the 2.0 version offers several revolutionary features, including Yet Another Resource Negotiator (YARN), HDFS Federation, and high availability, which make the Hadoop cluster much more efficient, powerful, and reliable.
The most serious limitations of classical MapReduce are primarily related to scalability, resource utilization, and the support of workloads different from MapReduce. In the MapReduce framework, the job execution is controlled by two types of processes: a single master process called JobTracker and a number of subordinate processes called TaskTrackers.
Apache Hadoop 2.0 includes YARN, which separates the resource management and processing components. The YARN-based architecture is not constrained to MapReduce. In YARN, MapReduce is simply degraded to a role of a distributed application (but still a very popular and useful one) and is now called MRv2. MRv2 is simply the re-implementation of the classic MapReduce engine, now called MRv1, which runs on top of YARN.
The course reviews MapReduce and provides insight into the design and implementation of YARN: ResourceManager instead of a cluster manager, ApplicationMaster instead of a dedicated and short-lived JobTracker, NodeManager instead of TaskTracker, a distributed application instead of a MapReduce job.
Course Syllabus
Lesson 1: Introduction to MapReduce and YARN
- Describe the MapReduce model v1 — this is the “classic” version that comes with Hadoop 1
- List the limitations of both Hadoop 1 and MapReduce 1
- Review the Java code required to handle the Mapper class, the Reducer class, and the program driver needed to access MapReduce
- Describe the YARN model, including the features of YARN and how a YARN program is run, and
- Compare “YARN / Hadoop 2 / MR2” versus “Hadoop 1 with MR1”
Lesson 2:Issues with/Limitations of Hadoop v1 & MapReduce v1
- List the limitations of MapReduce v1 and the need for MR v2 / YARN
- Describe MR2 / YARN processing
Lesson 3: The Architecture of YARN
- Understand the high level architecture of YARN
- Configuring, monitoring, and running applications in the YARN environment
Recommended skills prior to taking this course
- Know some basic Linux administration and commands
Grading scheme
- The minimum passing mark for the course is 60%, where the review questions are worth 40% and the final exam is worth 60% of the course mark.
- You have 1 attempt to take the exam with multiple attempts per question.
Requirements
None.
Course Staff
Glen R.J. Mules
Glen R.J. Mules is a Senior Instructor and Principal Consultant with IBM Information
Management World-Wide Education and works from New Rochelle, NY. Glen joined IBM in 2001
as a result of IBM’s acquisition of Informix Software. He has worked at IBM, and previously at
Informix Software, as an instructor, a course developer, and in the enablement of instructors
worldwide. He teaches courses in BigData (BigInsights & Streams), Optim, Guardium, and
DB2, & Informix databases. He has a BSc in Mathematics from the University of Adelaide, South
Australia; an MSc in Computer Science from the University of Birmingham, England; and has just
completed a PhD in Education (Educational Technology) at Walden University. His early work
life was as a high school teacher in Australia. In the 1970s he designed, programmed, and managed
banking systems in Manhattan and Boston. In the 1980s he was a VP in Electronic Payments for
Bank of America in San Francisco and New York.
In the early 1990s he was an EVP in Marketing for
a software development company and chaired the
ANSI X12C Standards Committee on Data Security
for Electronic Data Interchange (EDI).
Joe Byers
Joe Byers is a Senior Technical Curriculum
Developer with IBM World-Wide Education and
develops training on various media formats for
Business Intelligence, Predictive Analytics, and
Information Management. Joe came to IBM in
2007. Prior to that he was a Technical Manager for
Oracle Corporation, where he spent 10 years
engineering and architecting data warehouses. Joe
performed his undergraduate and graduate studies
in Computer Science at Indiana University, after
which, he became a database programmer for
Sears Craftsman Tools in Blue Ash, Ohio. In the
1990s, prior to joining Oracle, Joe was the Data
Administrator at The Andrew Jergens Company in
Cincinnati, Ohio, responsible for the entire
company’s data. While at Andrew Jergens, Joe
became an Oracle Master and earned his Oracle
Certified DBA (OCP). Joe regularly leads BI
sessions at IBM’s annual Insight conference in Las
Vegas. Joe’s other professional certifications
include five different certifications in IBM Cognos
products, two certifications in IBM TM1, as well as
certifications and badges in IBM DB2, SPARK,
Hadoop, and IBM Big Data.
User Reviews
Be the first to review “MapReduce and YARN”
Related Products
AdelaideX: Computational Thinking and Big Data
Learn the core concepts of computational thinking and how to collect, clean and consolidate large-scale datasets.
EdinburghX: Data Ethics, AI and Responsible Innovation
Our future is here and it relies on data. Medical robots, smart homes and cities, predictive policing, artificial intelligences – all are fuelled by data and all promise new benefits to society. But will these innovations benefit everyone? Who stands to gain and who is put at risk? How can we ensure that data is part of a just and sustainable world?
An Introduction to Spark
An introductory course for getting familiar with Spark and learning the basics of manipulating data using Spark.
AdelaideX: Big Data Analytics
Learn key technologies and techniques, including R and Apache Spark, to analyse large-scale data sets to uncover valuable business information.
HKUSTx: Foundations of Data Analytics
Learn the fundamental techniques for data analytics and to be prepared for learning and applying more advanced big data technologies.
IBM: Big Data, Hadoop, and Spark Basics
This course provides foundational big data practitioner knowledge and analytical skills using popular big data tools, including Hadoop and Spark. Learn and practice your big data skills hands-on.
There are no reviews yet.