Generate answers with mandatory citations and a refusal path - SkilBrill

LLM Application Development Training Generate answers with mandatory citations and a refusal path

LLM Application Development Training

0% Completed

M1 — The Shape of an LLM Application

An LLM app as a pipeline: retrieve, compose, call, validate, respond

Stateless calls versus conversational state and where history lives

The chat loop, message history and truncation strategy

Streaming responses and why time to first token changes the product

Framework choice: build on an SDK or adopt an orchestration framework

Failure modes that only appear once real users arrive

M2 — Retrieval and Chunking

Why chunking decides quality more than the model choice does

Fixed-size, semantic and structure-aware chunking compared

Overlapping windows, parent-child chunks and preserving context

Metadata, filters and restricting retrieval to the right source

Hybrid search: keyword plus vector and when each wins

Reranking: cross-encoder passes and the cost they are worth

M3 — Embeddings and Vector Stores

What an embedding captures and what it demonstrably does not

Choosing an embedding model and knowing when to re-index

Vector databases: indexes, distance metrics and approximate search

Scale, latency and storage behaviour at ten million chunks

Storing originals alongside vectors for citation and debugging

Re-indexing a corpus without breaking the live application

M4 — Generation, Tools and Structured Output

Composing the prompt from retrieved context and the user question

Function calling: exposing tools, schemas and argument validation

Multi-step tool use and when to stop calling tools

Structured output and validating it before it reaches the UI

Citations: tracing an answer back to the chunk it came from

Refusal and low-confidence paths when retrieval finds nothing

M5 — Production Engineering

Latency budgets: where the milliseconds actually go

Caching semantic results and when a cache key is safe

Model routing and prompt compaction to control spend

Concurrency, rate limits and provider fallbacks

Observability: tracing a request across retrieval and generation

Deploying, versioning and rolling back a prompt or index safely

M6 — Evaluation for RAG

Separating retrieval quality from generation quality

Building a question and gold-context set from real usage

Recall at k, precision and whether the right chunk was even fetched

Faithfulness: does the answer stay inside the retrieved context

End-to-end answer grading and the cost of each method

Regression testing an index change against a fixed benchmark

M7 — Applied Skills

Diagnosing a bad answer: retrieval problem or generation problem

Rebuilding a corpus ingestion job that silently dropped documents

Choosing between a vector store and a conventional database index

Writing an architecture decision record for an LLM component

Security review of an application that accepts user documents

Estimating build effort and monthly inference cost for a proposal

M8 — Capstone

Ingest a real document corpus with structure-aware chunking

Build hybrid retrieval with a reranking step and measure recall

Generate answers with mandatory citations and a refusal path

Add a function-calling tool for a live lookup and stream the response

Instrument tracing, cache the hot path and set a latency budget

Ship with a retrieval eval suite, runbook and rollback procedure

Announcements Course Info