Partitioning and shuffle tuning - SkilBrill

Skip to content

Home
Courses
About
Contact
Blog
- Freshers

Search for:

Home - Lessons - Data Engineering Training - M7 — Distributed Compute with Apache Spark - Partitioning and shuffle tuning

Data Engineering Training Partitioning and shuffle tuning

Data Engineering Training

0% Completed

M1 — Data Engineering Foundations

The data engineering lifecycle

Batch vs streaming trade-offs

Source-of-truth modelling concepts

Lab environment setup

M2 — SQL and Python for Data Engineers

Window functions and CTEs

Pandas and Polars essentials

PySpark in 90 minutes

Idempotent pipelines

M3 — Relational Modelling and Warehousing

Star schema and snowflake

Slowly Changing Dimensions (SCD2)

Surrogate keys and audit columns

dbt core models and snapshots

M4 — ETL and ELT Pipelines

Extract patterns: CDC, APIs, files

Transform with SQL, Spark, dbt

Load: warehouse, lake, lakehouse

Error handling and retries

M5 — Orchestration with Apache Airflow

DAGs, operators and sensors

Backfills and SLAs

Idempotency and retries

Production-grade Airflow on Kubernetes

M6 — Streaming with Apache Kafka

Topics, partitions, consumer groups

Exactly-once semantics

Schema Registry and Avro

Stream-table duality with Kafka Connect

M7 — Distributed Compute with Apache Spark

Spark architecture and DAGs

Structured Streaming APIs

Partitioning and shuffle tuning

Adaptive Query Execution

M8 — Lakehouse and Delta Lake

Medallion (bronze/silver/gold) architecture

Delta time travel and OPTIMIZE

Schema enforcement and MERGE

Unity Catalog basics

M9 — Data Quality and Observability

Great Expectations / Soda checks

Data contracts and lineage

Monitoring freshness, volume, quality

Alerting with PagerDuty / Slack

M10 — CI/CD and Infra-as-Code for Data

Git, CI pipelines and pull-request reviews

IaC with Terraform for data stacks

Schema migrations

Promoting dbt models across envs

M11 — Performance, Cost and Security

Query optimisation and Z-ordering

Cost tuning across Snowflake / Databricks

Row/column-level security

PII handling and masking

M12 — Capstone and Interview Prep

End-to-end lakehouse build

Failure-mode simulation

Behavioural and case interviews

Announcements Course Info

Global Trusted Software Training Partner Since 2022

SkilBrill Training Institute delivers industry-focused technology education that bridges the gap between academic learning and real-world demands. Our structured programs combine foundational theory with practical application, ensuring every graduate is confident and job-ready from day one.

Learn More About Us

© 2026 SkilBrill Training Institute. All Rights Reserved. Privacy Policy Terms Refunds Outcomes

📞 Enroll Now

Start Your Journey

Fill in your details and our team will get back to you within 24 hours

Please enable JavaScript in your browser to complete this form.

Full Name *

Email *

Course Interested In *

Mode of Course *