Simplifying data pipelines with Apache Kafka
|
Learn how to streamline your data pipelines using Apache Kafka. Explore its capabilities for real-time data streaming and integration with big data systems.
At a Glance
When you hear the terms, producer, consumer, topic category, broker, and cluster used together to describe a messaging system, something is brewing in the Kafka pipelines. Get connected and learn what that is, and what it means!
Many Big Data use cases have one thing in common – the use of Apache Kafka somewhere in the mix. Whether the distributed, partitioned, replicated commit log service is being used for messaging, website activity tracking, stream processing, or more, there’s no denying it is a hot technology. In this course, you will learn how Kafka is used in the real world and its architecture and components. You will quickly get up and running, producing and consuming messages using both the command line tools and the Java APIs. You also will get hands-on experience connecting Kafka to Spark, and working with Kafka Connect.
Course Syllabus
Lesson 1 – Introduction to Apache Kafka
- What Kafka is and why it was created
- The Kafka Architecture
- The main components of Kafka
- Some of the use cases for Kafka
Lesson 2 – Kafka Command Line
- The contents of Kafka’s /bin directory
- How to start and stop Kafka
- How to create new topics
- How to use Kafka command line tools to produce and consume messages
Lesson 3 – Kafka Producer Java API
- The Kafka producer client
- Some of the KafkaProducer configuration settings and what they do
- How to create a Kafka producer using the Java API and send messages both synchronously and asynchronously
Lesson 4 – Kafka Consumer Java API
- The Kafka consumer client
- Some of the KafkaConsumer configuration settings and what they do
- How to create a Kafka consumer using the Java API
Lesson 5 – Kafka Connect and Spark Streaming
- Kafka Connect and how to use a pre-built connector
- Some of the components of Kafka Connect
- How to use Kafka and Spark Streaming together
Prerequisites
- Have taken the Hadoop 101 course.
- Recommended skills prior to taking this course
- Basic understanding of Apache Hadoop and Big Data.
- Basic Linux Operating System knowledge.
- Basic understanding of the Scala, Python, R, or Java programming languages.
User Reviews
Be the first to review “Simplifying data pipelines with Apache Kafka”
Related Products
AdelaideX: Computational Thinking and Big Data
Learn the core concepts of computational thinking and how to collect, clean and consolidate large-scale datasets.
TokyoTechX: Introduction to Computer Science and Programming
The term “Computation” refers to the action performed by a computer. A computation can be a basic operation and it can also be a sophisticated computer simulation requiring a large amount of data and substantial resources. This course aims at introducing learners with no prior knowledge to the basic key concepts of computer science. By following the lectures and exercises of this course, you will gain an understanding of algorithms by programming using the language Ruby.
Introduction to Big Data and Hadoop
A beginner's guide to big data concepts and Hadoop framework, covering data storage, processing, and analysis techniques.
IBM: The Data Science Method
Learn about the methodology, practices and requirements behind data science to better understand how to problem solve with data and ensure data is relevant and properly manipulated to address a variety of real-world projects and business scenarios.
Mastering Big Data with Apache Spark and Java
Explore big data processing techniques using Apache Spark and Java, focusing on data analysis, manipulation, and real-time data processing.
Data Science in Production: Building Scalable Model Pipelines
Learn how to build scalable model pipelines for data science in production environments, focusing on best practices for deployment and management.
There are no reviews yet.