Building the evaluation set before building the feature - SkilBrill

Skip to content

Home
Courses
About
Contact
Blog
- Freshers

Search for:

Home - Lessons - AWS Generative AI Development - M6 — Evaluation, Safety and Guardrails - Building the evaluation set before building the feature

AWS Generative AI Development Building the evaluation set before building the feature

AWS Generative AI Development

0% Completed

M1 — Foundation Models and How They Behave

What a language model is predicting and why that matters

Model families on Bedrock: capability, context and cost

Temperature, top-p and the determinism question

Why models hallucinate and what does not fix it

Tokenisation, context windows and truncation

Choosing a model against a written requirement

M2 — Building on Amazon Bedrock

Account setup, model access and regional availability

The converse and invoke APIs and their differences

Streaming, stop reasons and error handling

Batch inference for cost on large workloads

Converse API conversation state and what to store

Structured output: JSON mode, schemas and parsing

M3 — Prompting and Structured Output

Prompt design as specification rather than magic wording

System prompts, roles and constraints that hold

Few-shot examples and when they stop helping

Output schemas and validation that rejects bad responses

Decomposing a hard task into a sequence of calls

Versioning prompts alongside application code

M4 — Retrieval-Augmented Generation

Why grounding a model on your documents changes the problem

Chunking strategy and what bad chunking does to answers

Embeddings, vector stores and Bedrock Knowledge Bases

Hybrid search and re-ranking for relevance

Citation, attribution and showing the source

Evaluating retrieval separately from generation

M5 — Agents and Tool Use

What an agent is and the loop it actually runs

Defining tools with schemas a model can choose between

Bedrock Agents, action groups and invocation roles

Constraining autonomy: budgets, steps and approvals

Failure handling when a tool call goes wrong

The cases where an agent is the wrong architecture

M6 — Evaluation, Safety and Guardrails

Building the evaluation set before building the feature

Measuring factuality, relevance, refusal and completeness

LLM-as-judge: how to use it without fooling yourself

Prompt injection: direct, indirect and document-borne

Guardrail content filters, topics and redaction

Human review loops and when they are unavoidable

M7 — Latency, Cost and Capacity

Token economics: input, output and cache pricing

Streaming versus waiting and the perceived latency trade-off

Prompt caching and context reuse

Provisioned throughput: when it pays and when it does not

Rate limits, quotas and graceful degradation

Setting a cost ceiling a feature cannot exceed

M8 — Capstone: A Deployed GenAI Feature

Take a written product brief and choose the model against it

Build a RAG feature over a real document set

Add tool use with a constrained, auditable action set

Write an evaluation suite and score the feature honestly

Install guardrails and a prompt-injection test

Ship it with monitoring, a budget alert and a rollback

Announcements Course Info

Global Trusted Software Training Partner Since 2022

SkilBrill Training Institute delivers industry-focused technology education that bridges the gap between academic learning and real-world demands. Our structured programs combine foundational theory with practical application, ensuring every graduate is confident and job-ready from day one.

Learn More About Us

© 2026 SkilBrill Training Institute. All Rights Reserved. Privacy Policy Terms Refunds Outcomes

📞 Enroll Now

Start Your Journey

Fill in your details and our team will get back to you within 24 hours

Please enable JavaScript in your browser to complete this form.

Full Name *

Email *

Course Interested In *

Mode of Course *