Why eyeballing ten examples proves nothing at scale - SkilBrill

Generative AI Training Why eyeballing ten examples proves nothing at scale

Generative AI Training

0% Completed

M1 — Foundations: How a Language Model Works

Transformers, attention and why the model predicts rather than recalls

Tokens: what gets counted, what gets priced and what gets truncated

Context windows, long-context behaviour and what falls out of view

Temperature, top-p and other sampling settings that change output

Determinism, seeds and why the same prompt gives two answers

Model families, size tiers and choosing a model for a task

M2 — Prompt Design as Engineering

The prompt contract: role, context, constraints, output format

Few-shot examples and choosing ones that actually shift behaviour

Structured output: JSON, schemas and enforcing a shape the code can parse

Chain-of-thought when reasoning helps, and when it just adds cost

System prompts, instructions hierarchy and precedence conflicts

Versioning a prompt like code, with diffs that can be reviewed

M3 — Evaluation and Measurement

Why eyeballing ten examples proves nothing at scale

Building an eval set from real traffic rather than imagined cases

Heuristic, model-based and human evaluation, and the cost of each

Scoring format compliance, factuality and instruction-following separately

Regression testing a prompt change against a fixed benchmark

Reading a score that went up while the product got worse

M4 — Grounding and Hallucination

Why a model invents a citation with total confidence

Grounding an answer in a retrieved source and requiring citations

Confidence thresholds and routing low-confidence answers to a person

Refusal behaviour: teaching a model to say it does not know

Factuality checks and claim-by-claim verification

Measuring hallucination rate before and after a change

M5 — Safety, Policy and Prompt Injection

Prompt injection: direct, indirect and document-borne attacks

System prompt exfiltration and why hidden instructions leak

Input and output filtering: what to block and what to log

PII, licensing and the data you must not send to a model

Writing an acceptable-use policy a reviewer can audit

Human-in-the-loop gates for high-consequence decisions

M6 — Cost, Latency and Deployment Choices

Pricing: input, output and cached tokens across providers

Cost modelling a feature at ten thousand requests a day

Latency: time to first token, streaming and perceived speed

Model routing: a small model for the easy cases, a large one for hard

Caching, batching and prompt compression to cut spend

Build versus buy: API, hosted endpoint or self-hosted weights

M7 — Applied Skills

Reading a failed generation and diagnosing prompt, model or data

Rewriting a vague business request into a testable specification

Comparing two models on your own eval set instead of a leaderboard

Turning a prototype notebook into something an engineer can review

Presenting AI risk to a non-technical stakeholder

Scoping a two-week pilot with a measurable success bar

M8 — Capstone

Pick a business process and define what the model must and must not do

Write the prompt contract and the structured output schema

Build an eval set from real examples and score the first version

Add grounding and citations, then measure the hallucination rate

Write the safety and injection policy plus the human escalation gate

Deliver a costed, tested proposal with a go or no-go recommendation

Announcements Course Info