Build the batch and streaming transformation layer - SkilBrill

Skip to content

Home
Courses
About
Contact
Blog
- Freshers

Search for:

Home - Lessons - Google Cloud Data Engineering - M8 — Capstone: End-to-End Data Platform - Build the batch and streaming transformation layer

Google Cloud Data Engineering Build the batch and streaming transformation layer

Google Cloud Data Engineering

0% Completed

M1 — The Google Cloud Data Platform

Workload classification and pipeline patterns

Storage classes and the lakehouse concept

BigQuery, Dataflow, Dataproc and Pub/Sub roles

Data residency, sovereignty and classification

Designing for repeatability and backfill

Reference architecture for a modern data platform

M2 — Ingestion: Batch and Streaming

Pub/Sub topics, subscriptions and acknowledgement

Exactly-once and at-least-once delivery semantics

Datastream for change data capture

Storage transfer and third-party transfers

Schema management and evolution

Backpressure and ingestion sizing

M3 — Batch Processing with Dataflow

Beam concepts: PCollection, transforms and pipelines

Batch versus streaming runners

Windowing, watermarks and triggers

Side inputs, joins and skew handling

Error handling and dead-letter design

Pipeline testing and template deployment

M4 — BigQuery Warehousing and Modelling

Dataset, table and partition design

Clustering, partitioning and the information schema

SQL performance: slots, joins and approximate queries

Star and snowflake modelling in BigQuery

Views, materialised views and semantic layers

BigQuery ML for in-warehouse modelling

M5 — Spark, Dataproc and the Lakehouse

Dataproc clusters, jobs and workflow templates

Spark on Cloud Storage versus HDFS

Delta and Iceberg table formats on GCS

Photon and performance tuning

Jupyter and interactive sessions

Cost control with autoscaling and preemptible nodes

M6 — Data Quality, Governance and Dataplex

Dataplex lakes, zones and assets

Data catalogue, glossary and lineage

Data quality rules and expectations

Profiling, classification and sensitive data discovery

Access control on governed data

Retention, deletion and audit evidence

M7 — Orchestration, CI/CD and Operations

Composer and workflow orchestration

Pipeline CI/CD and environment promotion

Monitoring pipelines with service level indicators

Data freshness and completeness monitoring

Incident response for broken pipelines

Cost monitoring and anomaly alerting

M8 — Capstone: End-to-End Data Platform

Capture streaming events into BigQuery

Build the batch and streaming transformation layer

Model the warehouse with partitioning and clustering

Apply quality rules and publish lineage

Orchestrate with SLAs and backfill support

Present the cost and reliability case

Announcements Course Info

Global Trusted Software Training Partner Since 2022

SkilBrill Training Institute delivers industry-focused technology education that bridges the gap between academic learning and real-world demands. Our structured programs combine foundational theory with practical application, ensuring every graduate is confident and job-ready from day one.

Learn More About Us

© 2026 SkilBrill Training Institute. All Rights Reserved. Privacy Policy Terms Refunds Outcomes

📞 Enroll Now

Start Your Journey

Fill in your details and our team will get back to you within 24 hours

Please enable JavaScript in your browser to complete this form.

Full Name *

Email *

Course Interested In *

Mode of Course *