Glue Data Catalog, Crawlers, and Classifiers - SkilBrill

AWS Data Engineering Training Glue Data Catalog, Crawlers, and Classifiers

AWS Data Engineering Training

0% Completed

Module 1 — AWS Data Engineering Foundations

AWS Global Infrastructure and IAM Fundamentals

Data Engineering Roles and Modern Data Architecture

AWS Analytics Services Overview: S3, Glue, Redshift, EMR, Kinesis

Lakehouse Architecture, Security, Networking, and Cost Management

Module 2 — SQL and Python for Data Engineering

Advanced SQL for Analytics and ETL

Python for Data Processing: pandas, boto3, psycopg2

Working with JSON, Parquet, CSV, and Avro

Coding Patterns, Unit Testing, and Modular Code Design

Module 3 — Amazon S3 and Data Lake

S3 Buckets, Objects, Versioning, and Lifecycle Policies

S3 Security: Bucket Policies, ACLs, and Encryption

Data Lake Design: Raw, Curated, Consumption Zones

Partitioning, File Formats, Athena, and Glue Data Catalog

Module 4 — AWS Glue and ETL Development

Glue Data Catalog, Crawlers, and Classifiers

Glue ETL Jobs with PySpark and Spark

Glue Studio Visual ETL

Dynamic Frames, Bookmarks, Workflows, and Triggers

Module 5 — Amazon Redshift Data Warehouse

Redshift Architecture, Distribution Styles, and Sort Keys

Loading Data from S3, Glue, and DMS

Query Optimization, WLM, and Concurrency Scaling

Spectrum, Redshift Serverless, and RA3 Nodes

Module 6 — Streaming Data with Kinesis and MSK

Kinesis Data Streams, Firehose, and Analytics

Apache Kafka on AWS with MSK

Stream Processing with Lambda and Kinesis Analytics

Windowing, Checkpointing, and Real-Time Analytics Architectures

Module 7 — AWS Lambda and Event-Driven Processing

Lambda Functions for Data Processing

Event Sources: S3, SQS, SNS, EventBridge, Kinesis

Step Functions for Orchestration

Error Handling, Dead-Letter Queues, and Lambda Optimization

Module 8 — Data Orchestration with MWAA and Step Functions

Amazon MWAA Managed Apache Airflow

Directed Acyclic Graphs (DAGs) Design

Step Functions State Machines for Data Pipelines

EventBridge Scheduling, Monitoring, and Alerting

Module 9 — Data Governance, Quality, and Security

AWS Lake Formation and Fine-Grained Access Control

Data Quality with Deequ and Great Expectations

Encryption at Rest and in Transit

VPC Endpoints, PrivateLink, and Compliance Frameworks

Module 10 — Big Data with EMR and Spark

EMR Architecture, Clusters, and Notebooks

Spark on EMR: DataFrames, SQL, Streaming

Hadoop Ecosystem on EMR

EMR Serverless, Managed Scaling, and S3/Glue/Redshift Integration

Module 11 — DevOps, CI/CD and Monitoring

Infrastructure as Code with CloudFormation and Terraform

CI/CD for Data Pipelines with CodePipeline

CloudWatch Monitoring, Logs, and Alarms

Cost Optimization, Disaster Recovery, and Backup Strategies

Module 12 — Real-World Projects and Interview Prep

E-Commerce Data Lake and Analytics Pipeline

Real-Time Clickstream Analytics with Kinesis

Data Warehouse Migration to Redshift

Portfolio, Project Explanation, and Interview Preparation

Announcements Course Info