arrow_back
Introduction to Data Engineering
Section Introduction
What is Data Engineering
Azure Data Factory Overview
Setup Your Environment
Creating Azure Free Account
Azure Portal Overview
Creating Azure Data Factory
Creating Azure Data Lake Storage Gen2
Creating Azure SQL Database
ADF Getting Started
Top Level Concepts in ADF
DataSet, Linked Services and Pipelines concept
Demo - How to create linked services, datasets and copy
Integration Runtime
Integration Runtimes (Compute for the Pipelines)
Integration Runtime types
Data Sources Location
Task: Self hosted integration runtime
Demo - Self Hosted Integration Runtime
Control Flow Activities
Control Flow Activities (Get Metadata , ForEach and IF)
Demo - Control Flow Activities (Get Metadata , ForEach and IF)
Delete Activity
Demo - Delete Activity
Stored Procedure and Lookup Activities
Demo - Transform Data Activities Stored Procedure and Lookup
Switch Activity
Demo - Switch Activity
Creating Dependency and Fail activity
Demo - Creating Dependency and Fail Activity
Wait
Demo - Wait
Nested Pipelines
Demo - Nested Pipelines
Until
Demo - Until Activity
Data Flow Activities
Data Flow - Select, Filter and Aggregate
Demo - Select, Filter and Aggeration DataFlow
Conditional Split
Demo - ConditionalSplit
Union
Demo - Union
Join
Demo - Join
Exists
Demo - Exists
Data Flow Lookup
Demo - Lookup DataFlow
Derived Column
Demo - Derived Column
Surrogate Key
Demo - Surrogate Key
Assert
Demo - Assert
AlterRow
Parse
Demo - Parse
Stringify
Demo - Stringify
Window
Demo - Window
Pivot
Demo - Pivot
Unpivot
Demo - Unpivot
Flowlets
Demo - Flowlet
Flatten
Demo - Flatten
Parameterize
Parameterizing the Pipelines
Parameterization of Pipelines
Demo - Parametrizing Pipelines
Triggers
Triggers and Types of Triggers
Scheduled Trigger
Demo - Schedule Trigger
Storage Event Trigger
Demo - Storage Event Trigger
Tumbling Trigger
Demo - Tumbling Trigger
Auditing
Auditing
Demo - Auditing
Securing ADF Credentials
Security Azure Key Vaults
Demo - Security Azure Key Vaults
Networking for Azure IR
Networking for ADF and Synapse Azure provided IR
Demo - Networking Azure Provided IR
Development Flow (DevOps)
Development flow using DevOps
Demo - DevOps Pipelines
Sharing SHIR
Sharing Self hosted Integration Runtime
Demo - Shared Self Hosted Integration Runtime
Monitoring
Monitor and Manage