Incremental Load and CDC Patterns in ADF - SkilBrill

Azure Data Engineering Training in Gurgaon Incremental Load and CDC Patterns in ADF

Azure Data Engineering Training in Gurgaon

0% Completed

Data Engineering Foundations and Architecture

Introduction to Data Engineering and the Modern Data Stack

Data Pipeline Components: Ingestion, Storage, Transformation, Serving

Batch vs Streaming vs Lambda Architecture

Cloud Data Engineering on Azure: Overview and Roadmap

SQL for Data Engineering

Advanced SQL: Joins, Window Functions, CTEs

Data Cleaning and Transformation with SQL

Indexing, Partitioning, and Query Optimization

SQL Interview Patterns for Data Engineers

Python for Data Engineering

Python Essentials for Data Engineering

Working with APIs, Files, and Pandas

PySpark Basics and DataFrame Operations

Python Interview Patterns and Coding Practice

Azure Data Lake Storage Gen2

ADLS Gen2 Architecture and Hierarchical Namespace

Container, Directories, and File Management

Security: ACLs, RBAC, and Shared Access Signatures

Integrating ADLS Gen2 with ADF and Databricks

Azure Data Factory

ADF Pipelines, Datasets, Linked Services, and Integration Runtime

Data Movement and Transformation Activities

Parameterization, Dynamic Expressions, and Metadata

Incremental Load and CDC Patterns in ADF

Azure Synapse Analytics

Synapse Workspace, SQL Pools, and Spark Pools

Dedicated SQL Pool: Tables, Indexing, and Distribution

Synapse Pipelines and Data Flows

Serverless SQL and External Tables over Data Lake

Azure Databricks and Apache Spark

Databricks Workspace, Clusters, and Notebooks

Spark Architecture: RDDs, DataFrames, and Catalyst Optimizer

Delta Lake: ACID, Time Travel, and Optimize

Databricks Unity Catalog and Governance

Microsoft Fabric, Data Warehousing, and Data Modelling

Microsoft Fabric: Lakehouse, Warehouse, and OneLake

Data Warehouse Design: Star Schema, Snowflake Schema

Slowly Changing Dimensions and Surrogate Keys

Dimensional Modeling Best Practices

ETL, ELT, CDC, and Metadata-Driven Pipelines

ETL vs ELT: When to Use Which

Change Data Capture with Azure SQL and Debezium Patterns

Metadata-Driven Pipeline Design

Incremental Data Loading Strategies

Batch and Streaming Data Engineering

Batch Processing Patterns and Scheduling

Streaming Concepts: Windowing, Watermarking, Checkpointing

Azure Event Hubs: Producers, Consumers, and Partitions

Spark Structured Streaming on Azure

Security, Governance, Quality, and Observability

Azure RBAC, Managed Identity, and Azure Key Vault

Data Security: Encryption, Masking, and PII Handling

Data Quality Frameworks and Great Expectations

Data Observability and Governance with Microsoft Purview

DevOps, Monitoring, Performance, and Cost Optimization

Git, Branching Strategies, and Code Reviews

CI/CD with Azure DevOps for Data Pipelines

Pipeline Monitoring, Alerting, and Troubleshooting

Performance Tuning and Cost Optimization on Azure

Real-World Projects and Production Architecture

Retail Data Engineering Pipeline: End-to-End Build

Customer 360 Data Platform: Design and Implementation

Real-Time IoT Data Pipeline with Event Hubs and Spark

Metadata-Driven Enterprise Pipeline Framework

Interview Preparation and Career Development

Azure Data Engineering Interview Question Bank

SQL and Python Interview Rounds

Project Explanation and Portfolio Building

Resume, LinkedIn, and Job Search Strategy

Announcements Course Info