RAG on AWS Bedrock: Build a Retrieval-Augmented Generation Pipeline Step by Step

Quick Answer: A RAG pipeline on Amazon Bedrock has five stages: ingest documents, chunk them, embed and store vectors (usually OpenSearch Serverless or pgvector), retrieve relevant chunks at query time, and generate an answer with a Bedrock model grounded in those chunks. The hardest part is not the model — it is chunking strategy and evaluation.

Why RAG Exists at All

Large language models have a fixed training cutoff and no knowledge of your private data. Fine-tuning is often suggested as the alternative, but for most business use cases it is the wrong tool — it is expensive, slow to update, and does not reliably teach a model specific facts. RAG solves both problems by retrieving relevant documents at query time and passing them to the model as context.

The result is an answer grounded in your actual data, with sources you can cite — which matters enormously in regulated industries where “the model said so” is not an acceptable answer.

The Architecture

Documents (S3)
    │
    ▼
Ingestion ──► Chunking ──► Embedding (Titan / Cohere)
    │                                          │
    ▼                                          ▼
Metadata store (DynamoDB)          Vector index (OpenSearch)
                                                │
User query ──► Embed ──► Retrieve top-k ────────┤
                                                ▼
                                    Bedrock model (Claude / Nova)
                                                │
                                                ▼
                              Grounded answer + citations

Stage 1: Ingestion and Chunking

Chunking is where most RAG pipelines quietly fail, and it is the stage people spend least time on. Chunk too large and retrieval becomes imprecise — you pull 4,000 tokens of which 400 are relevant, diluting the context. Chunk too small and you lose the surrounding meaning, retrieving a sentence that does not make sense alone.

Practical starting points that work:

• 500–800 tokens with 10–20% overlap is a reasonable default for most documents
• Split on structure first — headings, paragraphs, list items — before falling back to fixed sizes
• Never split mid-sentence or mid-table if you can avoid it; broken context retrieves badly
• Carry metadata forward — document title, section heading, date — onto every chunk

That last point matters more than people expect. When you later ask “which document did this come from,” you need the metadata attached to the chunk, not looked up afterwards.

Stage 2: Embedding and Storage

Amazon Bedrock offers Amazon Titan Embeddings and Cohere Embed models. Titan is the default choice for most AWS-native setups; Cohere often performs better on multilingual content, which matters for Indian enterprises with mixed-language documents.

For the vector store, you have three realistic options on AWS:

OpenSearch Serverless deserves particular attention because it supports hybrid search — combining vector similarity with keyword (BM25) matching. Pure vector search misses exact identifiers: account numbers, error codes, part numbers. Hybrid search catches both, and in practice it meaningfully improves retrieval quality on enterprise document sets.

Stage 3: Retrieval

Retrieval looks simple — embed the query, find the nearest chunks — but the details decide whether your system works.

Top-k selection. More chunks is not better. Retrieving 20 chunks and passing them all to the model usually produces worse answers than retrieving 4–6 good ones, because irrelevant context confuses the model and inflates cost. Tune this against your evaluation set rather than guessing.

Reranking. For higher accuracy, retrieve a wider set (say top-20) then rerank with a dedicated model, passing the top 4–5 to generation. Bedrock offers Cohere Rerank for this. It adds latency and cost but substantially improves precision on hard queries.

Stage 4: Generation With Grounding

The prompt structure that works reliably:

You are answering questions about company policy documents.
Answer ONLY from the context below. If the context does not
contain the answer, say you don't have that information —
do not guess.

Context:
{retrieved_chunks}

Question: {user_query}

Cite the source document for each claim.

Two elements do the real work. The explicit instruction not to guess when the context is insufficient — without it, models confidently hallucinate around gaps. And the citation requirement, which both improves answer quality and gives users something verifiable.

On model choice: Claude models on Bedrock are the common production choice for their instruction-following and long-context handling. Amazon Nova models are worth evaluating for cost, particularly at volume. Run both against your own evaluation set rather than trusting benchmark claims.

Stage 5: Evaluation — the Part Everyone Skips

This is the most important section of this guide and the one most often ignored. Without evaluation, you cannot tell whether a change helped or broke your pipeline, so you end up tuning by vibes.

Build a small evaluation set first — even 30–50 question-and-answer pairs with expected sources. Then measure two things separately:

• Retrieval quality — did we retrieve the right chunks? Measured by whether the correct document appears in the retrieved set.
• Answer quality — given correct retrieval, is the answer accurate and grounded? Measured by faithfulness to the context.

Separating these matters because the failure modes differ. If retrieval is broken, no prompt engineering will fix the answers. If retrieval is fine but answers are poor, the problem is in generation. Teams that measure only the end-to-end answer cannot diagnose which is failing.

Production Concerns Nobody Mentions in Tutorials

Cost control. Token spend grows quickly. Cache frequent queries, cap chunk counts, and use smaller models for retrieval-adjacent tasks. Set up cost alerting before launch, not after the first bill.

Freshness. Documents change. You need a re-ingestion strategy — incremental updates triggered on document change, not a full rebuild, which becomes prohibitively slow at scale.

Access control. If documents have different visibility rules, retrieval must respect them. Filtering at the vector store by user permissions is far better than post-retrieval filtering, which leaks information through ranking.

Training Resources

Build this end to end with SkilBrill AWS Generative AI Development Training. The ML foundations behind embedding and evaluation are covered in AWS Machine Learning Training.

FAQ

What is RAG in Amazon Bedrock?

Retrieval-Augmented Generation: you retrieve relevant documents at query time and pass them to a Bedrock model as context, so answers are grounded in your private data rather than the model’s training knowledge.

Which vector database should I use with Bedrock?

OpenSearch Serverless for most production use — it supports hybrid search, which improves retrieval on enterprise documents. pgvector on Aurora if your team already runs Postgres. Both are solid choices; the evaluation set matters more than the store.

What chunk size works best for RAG?

500–800 tokens with 10–20% overlap is a good starting point. Split on document structure before using fixed sizes, and always carry metadata forward onto each chunk. Tune against your own evaluation data.

Why are my RAG answers inaccurate?

Usually retrieval, not generation. Check whether the correct documents are being retrieved at all before adjusting prompts. Separate retrieval evaluation from answer evaluation so you can diagnose which stage is failing.

Written by SkilBrill Training Institute

October 15, 2026 · 6 min read