Use Spark Notebooks in an Azure Synapse Pipeline
|
Apache Spark provides data engineers with a scalable, distributed data processing platform, which can be integrated into an Azure Synapse Analytics pipeline.
Describe notebook and pipeline integration., Use a Synapse notebook activity in a pipeline., Use parameters with a notebook activity.
Prerequisites
Before starting this module, you should have experience of using Apache Spark in Azure Synapse Analytics. Consider completing the Analyze data with Apache Spark in Azure Synapse Analytics module first:
User Reviews
Be the first to review “Use Spark Notebooks in an Azure Synapse Pipeline”
Related Products
Data lakes and Lakehouses with Spark and Azure Databricks
Learn to manage and process data using Spark and Azure Databricks platforms.
Load data into a relational data warehouse
A core responsibility for a data engineer is to implement a data ingestion solution that loads new data into a relational data warehouse.
Perform data engineering with Azure Synapse Apache Spark Pools
Apache Spark is a highly scalable distributed processing solution for big data analytics and transformation. You can leverage its power in Azure Synapse Analytics by using Spark pools.
Use Apache Spark in Azure Databricks
Azure Databricks is built on Apache Spark and enables data engineers and analysts to run Spark jobs to transform, analyze and visualize data at scale.
Transfer and transform data with Azure Synapse Analytics pipelines
Azure Synapse Analytics enables data integration through the use of pipelines, which you can use to automate and orchestrate data transfer and transformation activities.
Use Azure Synapse serverless SQL pool to query files in a data lake
With Azure Synapse serverless SQL pool, you can leverage your SQL skills to explore and analyze data in files, without the need to load the data into a relational database.
There are no reviews yet.