Train, validation and test splits that do not leak - SkilBrill

Skip to content

Home
Courses
About
Contact
Blog
- Freshers

Search for:

Home - Lessons - AWS Machine Learning - M1 — Framing ML Problems on AWS - Train, validation and test splits that do not leak

AWS Machine Learning Train, validation and test splits that do not leak

AWS Machine Learning

0% Completed

M1 — Framing ML Problems on AWS

What is genuinely an ML problem and what is a rules problem in disguise

Prediction types: regression, classification, ranking, forecasting

Writing the target, the metric and the baseline before modelling

Train, validation and test splits that do not leak

Where SageMaker fits in the wider AWS data estate

Killing an ML project that will not clear its own metric

M2 — Data Preparation and Feature Engineering

Ingestion and preparation with Glue and SageMaker Data Wrangler

Cleaning, missing values and category handling

Feature engineering patterns that generalise

Building a feature store and keeping it consistent

Class imbalance, sampling and what it costs you later

Reproducible datasets and lineage

M3 — SageMaker Training and Algorithms

Built-in algorithms and when to use each one

Notebook instances, training jobs and the estimator model

Hyperparameter tuning jobs and early stopping

GPU versus instance selection and training cost control

Training on your own container when built-ins do not fit

Recording experiment results so comparisons survive

M4 — Deployment and Inference

Real-time endpoints versus batch transform

Model packaging, inference scripts and dependencies

Variant deployment, traffic splitting and shadow tests

Autoscaling and latency targets for a serving path

A/B and canary rollout with a rollback you have rehearsed

Inference cost modelling before the endpoint goes live

M5 — Monitoring, Drift and Retraining

What to monitor: latency, errors, and the model itself

Data drift and concept drift with CloudWatch Model Monitor

Ground-truth capture and delayed label problems

Setting drift thresholds that do not cry wolf

Triggering retraining on evidence rather than a schedule

What to do when the model quietly degrades

M6 — MLOps: Pipelines and Model Registry

SageMaker Pipelines for end-to-end orchestration

Steps, parameters, caching and run reproducibility

Model registry, versioning and approval gates

CI/CD for ML: tests that include data and model checks

Metadata, lineage and experiment tracking with MLflow

Promotion between environments without retraining drift

M7 — Security, Governance and Model Risk

IAM boundaries for training and inference roles

Data privacy and DPDP obligations on training data

Model explainability and attribution techniques

Bias evaluation and documenting what you tested

Audit evidence: what a regulator or client will ask for

When a model should be retired rather than retrained

M8 — Capstone: An End-to-End ML System

Frame a business problem as an ML task

Prepare and validate the training data with leakage checks

Train a SageMaker model and beat the stated baseline

Deploy it to a monitored endpoint with traffic splitting

Build the retraining pipeline with lineage in the registry

Present the model, its limits and its risk position

Announcements Course Info

Global Trusted Software Training Partner Since 2022

SkilBrill Training Institute delivers industry-focused technology education that bridges the gap between academic learning and real-world demands. Our structured programs combine foundational theory with practical application, ensuring every graduate is confident and job-ready from day one.

Learn More About Us

© 2026 SkilBrill Training Institute. All Rights Reserved. Privacy Policy Terms Refunds Outcomes

📞 Enroll Now

Start Your Journey

Fill in your details and our team will get back to you within 24 hours

Please enable JavaScript in your browser to complete this form.

Full Name *

Email *

Course Interested In *

Mode of Course *