> study notes · amazon applied scientist ii

RAG Architecture & Evaluation

Comprehensive notes covering retrieval-augmented generation end-to-end: pipeline architecture, chunking strategies, embedding models, vector databases, retrieval methods, evaluation metrics, failure modes, and production considerations.

• RAG
• Embeddings
• Vector DBs
• BM25
• NDCG
• RAGAS
• Agentic RAG
• Production ML

Why LLMs Hallucinate: 3-Point Summary

What is RAG and Why It Exists

Retrieval-Augmented Generation (RAG) is a framework that enhances LLMs by grounding their outputs in externally retrieved documents. Instead of relying purely on parametric knowledge baked into model weights during training, RAG retrieves relevant context at inference time and injects it into the prompt.

The Core Problem RAG Solves

• →Knowledge cutoff: LLMs have a training cutoff date and cannot answer questions about recent events without retrieval.
• →Hallucination: LLMs generate plausible-sounding but factually incorrect text. Retrieved documents ground the answer in real sources.
• →Domain specificity: General-purpose LLMs lack proprietary or niche domain knowledge (e.g., sustainability emissions factors, internal policy documents).
• →Auditability: RAG outputs can be traced back to source documents — critical for sustainability claims verification.

RAG Generations

RAG vs Fine-tuning

One of the most common interview questions in applied LLM roles. The answer isn't either/or — but knowing when each is the right tool matters.

RAG Pipeline Architecture — End to End

A production RAG pipeline has two distinct phases: offline indexing and online retrieval-generation.

Chunking Strategies

Chunking is one of the highest-impact decisions in RAG. Too large = diluted relevance signal, context overflow. Too small = loses context, incomplete answers.

Strategy Comparison

Embedding Models

Embedding models convert text into dense vectors. The quality of embeddings directly determines retrieval quality — garbage embeddings mean garbage retrieval, no matter how good your LLM is.

Evolution of Text Embeddings

• →TF-IDF / BM25 (sparse): Bag-of-words representation. High-dimensional sparse vectors. No semantic understanding — 'car' and 'automobile' are unrelated.
• →Word2Vec / GloVe (dense, word-level): Each word gets a fixed vector learned from co-occurrence statistics. Cannot handle polysemy ('bank' = riverbank or financial bank?). No context-sensitivity.
• →BERT-based (contextual, token-level): Each token gets a vector that depends on surrounding context. Handles polysemy. But designed for classification, not retrieval.
• →Sentence Transformers / SBERT: Fine-tuned BERT with siamese/triplet network to produce sentence-level embeddings optimized for semantic similarity. State of the art for RAG retrieval.
• →OpenAI text-embedding-3: API-based, 1536 or 3072 dimensions, strong performance. No control over training, cost per token.

Bi-Encoder vs. Cross-Encoder

Contrastive Learning Objectives

• →InfoNCE loss: For a positive pair (query, relevant doc), maximize similarity relative to N negative pairs. Softmax over all pairs. Used in DPR, E5.
• →Triplet loss: Anchor + positive + negative. Minimize distance(anchor, positive) minus distance(anchor, negative) + margin.
• →SimCLR: Self-supervised contrastive learning. Augmentations of the same input are positive pairs. Useful for multimodal embeddings.

Similarity Metrics

Dimensionality Reduction

• →PCA: Linear projection onto principal components. Preserves global variance. Fast but loses non-linear structure.
• →UMAP: Non-linear. Preserves local neighborhood structure better than t-SNE. Good for visualization and approximate compression.
• →Matryoshka embeddings: Train embeddings so that the first D dimensions are a good representation — allows truncating at inference time for speed/storage trade-offs.

Vector Databases — Trade-offs

Vector databases are purpose-built for Approximate Nearest Neighbor (ANN) search over dense embeddings. The key trade-off in all of them: recall vs. speed vs. memory.

Core ANN Index Types

• →HNSW (Hierarchical Navigable Small World): Graph-based index. High recall, fast queries, but high memory usage. Default in Weaviate, Qdrant, Milvus. Best general-purpose choice.
• →IVF (Inverted File Index): Cluster-based. Divides space into Voronoi cells. Fast build, lower memory. Must tune nprobe (how many clusters to search). Used in FAISS.
• →PQ (Product Quantization): Compresses vectors by quantizing sub-spaces. 8–32x compression. Combined with IVF as IVF+PQ for large-scale deployment. Lower recall.
• →Flat Index: Exact brute-force search. Perfect recall. Only viable for small corpora (< 1M vectors).

Vector Database Comparison

Key Parameters to Tune

• →ef_construction (HNSW): Controls index build quality. Higher = better recall, slower build.
• →ef (HNSW query): Controls search quality at query time. Higher = better recall, slower query.
• →nprobe (IVF): Number of clusters searched per query. Higher = better recall, slower.
• →m (HNSW): Number of connections per node. Higher = better recall, more memory.

ANN Algorithms — How Vector Search Actually Works

Every vector database under the hood uses an Approximate Nearest Neighbor (ANN) index. Understanding these algorithms is essential for production RAG — they control the recall/speed/memory triangle that governs retrieval performance.

Why ANN Exists — The Scaling Problem

The naive approach — compare the query against every stored vector — is called exact nearest neighbor search.

The Core Trade-off Triangle

KD-Trees — Space Partitioning

Recursively partitions the vector space by splitting on one dimension at a time, building a binary tree. At query time, traverse the tree and prune branches that cannot possibly contain a nearer neighbor.

• →Building: At each node, split on the dimension with highest variance. Recursively partition left/right subtrees until leaf size is small.
• →Pruning rule: Skip a subtree if the distance from the query to the split hyperplane is greater than the current best distance.

Ball Trees — Hypersphere Partitioning

Like KD-trees but partition space into hyperspheres instead of hyperplanes. Handles non-axis-aligned clusters better and works in slightly higher dimensions.

• →Building: Each node defines a ball: (center=centroid, radius=max distance to any point). Split by picking two furthest seeds, assign points to nearest seed, recurse.
• →Pruning rule: Skip a ball if: dist(query, center) - radius > best_dist_so_far. I.e., even the closest possible point in the ball can't beat the current best.

LSH — Locality Sensitive Hashing

Design hash functions where similar vectors are more likely to produce the same hash value. Opposite of cryptographic hashes which deliberately make similar inputs produce different hashes.

• →Random Projection LSH: Project vectors onto random unit vectors. Similar vectors land near each other on the line. Divide into buckets — similar vectors fall into the same bucket.
• →K hash functions per table: Use K hash functions and require all K to agree. Higher K → fewer false positives but more false negatives (similar vectors split across bucket boundaries).
• →L tables: Repeat with L independent sets of K hash functions. At query time, look up in all L tables and union candidates. Higher L → better recall, more memory.

IVF — Inverted File Index (KMeans-Based)

Divide the vector space into k clusters using KMeans. Each vector is assigned to its nearest centroid. At query time, find the nearest cluster centroids and only search vectors within those clusters.

• →Building: Run KMeans with k clusters on all N vectors. Build inverted lists: inverted_list[i] = all vectors assigned to cluster i. Build time: O(N × k × D × iterations).
• →nprobe: The main recall-speed knob. Controls how many clusters you search per query. nprobe=1 → fastest, ~80-90% recall. nprobe=k → exact search (same as brute force).

HNSW — Hierarchical Navigable Small World

The best general-purpose ANN algorithm for production. Used by Weaviate, Qdrant, Milvus, and FAISS. Builds a multi-layer graph where upper layers have sparse long-range connections for fast navigation, lower layers have dense short-range connections for precise search.

• →Layer assignment: When inserting a vector, randomly assign its max layer using exponential decay: max_layer = floor(-ln(random()) × mL). ~63% of vectors land only in Layer 0. A vector appears in ALL layers 0 to max_layer.
• →Upper layers: Express highways — sparse, long-range connections. Lets you navigate to the right region of the space in a few hops.
• →Layer 0: Local streets — dense, short-range connections. Precise search within the right neighborhood.
• →ef parameter: Controls how many candidates you track during search. ef=50 is typical production starting point. ef must be ≥ k. Tune this first — it's the main production knob.

Algorithm Comparison

Retrieval Strategies

Sparse Retrieval — TF-IDF and BM25

Sparse retrieval represents documents as high-dimensional sparse vectors (one dimension per vocabulary term). No ML required.

• →TF-IDF: Term Frequency × Inverse Document Frequency. Upweights rare terms that are specific to a document. Downweights common words. Simple but no length normalization.
• →BM25: The dominant sparse retrieval model. Extends TF-IDF with saturation (TF gains diminish as term frequency increases) and length normalization. Parameters: k1 (TF saturation, typically 1.2–2.0) and b (length normalization, typically 0.75).

Strength: Exact keyword matching — critical when domain has specialized terminology (emissions factor codes, regulatory identifiers, chemical names). Weakness: No semantic understanding. “CO₂ emissions” and “carbon footprint” are unrelated to BM25.

Dense Retrieval

Embed both query and documents into the same dense vector space. Retrieve by nearest neighbor search. Captures semantic similarity that sparse methods miss.

• →DPR (Dense Passage Retrieval): Two separate BERT encoders — one for questions, one for passages — trained with contrastive loss on QA pairs.
• →ColBERT: Late interaction model. Stores one vector per token. At query time, computes MaxSim — max similarity between each query token and all document tokens. Higher recall than single-vector methods but larger index.

Hybrid Retrieval

Combine sparse (BM25) and dense (embedding) retrieval for best of both worlds.

• →Reciprocal Rank Fusion (RRF): Combine rankings from multiple retrieval systems without needing score normalization. score(d) = Σ 1/(k + rankᵢ(d)) where k=60 is a smoothing constant. Simple and effective.
• →Linear combination: Weighted sum of normalized BM25 and cosine similarity scores. Requires careful normalization and weight tuning.

Re-ranking

• →Cross-encoder re-ranking: Concatenate query + document, run through BERT. Produces a single relevance score. O(K) inference calls at query time. High quality but slow.
• →ColBERT re-ranking: More efficient late-interaction. Precompute token embeddings offline, do MaxSim at query time.
• →LLM re-ranking: Use the LLM itself to judge relevance. Highest quality, highest latency.

Query Expansion

• →HyDE (Hypothetical Document Embeddings): Use the LLM to generate a hypothetical answer to the query, then retrieve using that answer. The hypothesis is often closer to real documents in embedding space than the original question.
• →Multi-query: Generate multiple paraphrases of the query. Retrieve for each. Union the results. Improves recall.
• →Step-back prompting: Generalize the specific query to a broader question before retrieval. Useful when the question is too specific to match documents.

Information Retrieval Evaluation Metrics

Be able to compute NDCG by hand. These metrics are standard in IR and search evaluation, and directly apply to measuring RAG retrieval quality.

RAG Evaluation — RAGAS Framework

Evaluating RAG requires assessing both retrieval quality AND generation quality. RAGAS (RAG Assessment) is the standard framework for end-to-end evaluation.

The Four Core RAGAS Metrics

RAG Failure Modes and How to Fix Them

Advanced RAG Techniques

Advanced RAG Concepts — Deep Dives

Three techniques that separate production-grade RAG from toy demos: multi-hop retrieval for questions that span multiple documents, structured knowledge handling for tables and relationships, and adaptive strategies that route queries to the right pipeline.

1 · Multi-Hop and Cross-Document Retrieval

Standard RAG works when the answer lives in one chunk. Multi-hop handles questions that require chaining facts across multiple documents — each retrieval step informs the next query.

Query: "What is the emissions intensity of Amazon's top electronics supplier?"

Step 1: Retrieve for original query
  → "Foxconn is Amazon's largest electronics supplier"

Step 2: Reformulate with what was found
  → new query: "Foxconn annual emissions"
  → "Foxconn emitted 9.2M tonnes CO2e in 2023"

Step 3: Reformulate again
  → new query: "Foxconn annual revenue 2023"
  → "Foxconn revenue was $214B in 2023"

Step 4: Generate final answer
  → emissions intensity = 9.2M / 214B = 43 tonnes per $M revenue

IRCoT (Interleaved Retrieval + Chain of Thought):
  Interleave reasoning steps with retrieval steps.
  Reason: "I need the top supplier first"
  Retrieve: "Foxconn is top supplier"
  Reason: "Now I need Foxconn's emissions"
  Retrieve: "9.2M tonnes CO2e"
  Reason: "Now I need revenue to compute intensity"
  Retrieve: "$214B revenue"
  Generate: "Emissions intensity = 43 tonnes per $M"

ReAct pattern:
  Thought → Action (retrieve) → Observation → Thought → ...
  Loop until sufficient information to answer.

2 · Table Extraction and Graph RAG

Standard RAG embeds text chunks. But huge amounts of sustainability data live in tables and relationship graphs — structures that text embeddings can't represent well.

Table Extraction

Raw table embedded as text (bad):
"Scope 1 2021 45000 Scope 2 2021 123000 Scope 3 2021 890000
 Scope 1 2022 42000 Scope 2 2022 118000..."

The embedding has no idea this is a 2D structure.
Rows and columns that have meaning are flattened away.

• →Table-to-text verbalization: Convert each cell to natural language before embedding. "In 2021, Scope 1 emissions were 45,000 tonnes. In 2022, Scope 1 decreased to 42,000 tonnes." The embedding now captures meaning.
• →Row-per-chunk: Each row becomes its own chunk with column headers prepended. "Year: 2021 | Category: Scope 1 | Value: 45,000 tonnes CO2e"
• →SQL / structured retrieval: Parse the table into a database. Convert natural language questions to SQL at query time. Most reliable for numerical questions and comparisons.
• →Multimodal embedding: Embed the table as an image. Models like GPT-4V can reason about table structure visually — useful when tables have complex merged cells or visual formatting.

Graph RAG

Standard RAG treats documents as isolated chunks with no explicit connections. Graph RAG builds a knowledge graph where entities are nodes and relationships are edges — then uses graph traversal for retrieval.

Standard RAG index (disconnected chunks):
  Chunk 1: "Amazon partners with Rivian for electric delivery vans"
  Chunk 2: "Rivian produces zero-emission vehicles"
  Chunk 3: "Amazon's Scope 3 transport emissions fell 8% in 2023"
  → No explicit link between them.

Graph RAG index (explicit relationships):
  [Amazon] --PARTNERS_WITH--> [Rivian]
  [Rivian] --PRODUCES--> [Electric Delivery Vans]
  [Electric Delivery Vans] --REDUCES--> [Scope 3 Emissions]
  [Amazon] --HAS--> [Scope 3 Emissions]

Query: "How is Amazon reducing transport emissions?"
  → Graph traversal: Amazon → partners → Rivian
    → produces → electric vans → reduces → Scope 3

Microsoft GraphRAG (2024)

• →Extracts entities and relationships from all documents using an LLM
• →Builds a community structure — clusters of related entities
• →Generates summaries for each community
• →At query time uses both graph traversal and vector search

3 · Adaptive Retrieval Strategies

A fixed pipeline treats every query the same way. Adaptive retrieval detects the query type and routes it to the right strategy — avoiding wasted retrieval for simple questions and under-powered retrieval for complex ones.

Query A: "What is CO2?"
→ General knowledge. LLM already knows this.
  Retrieval wastes time and adds noise. Don't retrieve.

Query B: "What were our Scope 3 emissions in Q3 2024?"
→ Specific factual lookup. Needs retrieval.
  Dense retrieval is fine — straightforward semantic match.

Query C: "Compare emissions reduction strategies across
          all suppliers in Southeast Asia"
→ Multi-document synthesis. Needs multi-hop retrieval.

Query D: "Verify this supplier's claimed 30% reduction"
→ Targeted evidence retrieval + citation checking.

A fixed pipeline applies the same strategy to all four.

Step 1: Query classification
  - Does this need retrieval at all?
  - Is it simple (single-hop) or complex (multi-hop)?
  - Is it a lookup, comparison, or verification task?

Step 2: Route to the right strategy
  No retrieval needed  → straight to LLM
  Simple factual       → dense retrieval, top-3 chunks
  Complex multi-part   → multi-hop retrieval pipeline
  Comparison           → retrieve from multiple sources, synthesize
  Verification         → targeted retrieval + citation extraction

Step 3: Adjust K dynamically
  Simple query  → K=3 (less context needed)
  Complex query → K=10 (more context needed)

Step 4: Confidence-based fallback
  Retrieval confidence low → trigger web search (CRAG)

Self-RAG — the model decides when to retrieve

The model learns to emit special tokens:
  [Retrieve]    → "I need to retrieve for this"
  [No Retrieve] → "I already know this, skip retrieval"
  [IsRel]       → "Is this retrieved chunk relevant?"
  [IsSup]       → "Does this chunk support my answer?"
  [IsUse]       → "Is this retrieval useful overall?"

The model scores its own retrievals and decides whether
to use them, retry, or generate without retrieval.

Production RAG Considerations

Latency vs. Throughput Trade-offs

Caching

• →Query embedding cache: Cache query embeddings for repeated queries. LRU cache with TTL.
• →Semantic cache: Cache (query_embedding → answer) pairs. On new query, check if a semantically similar query was answered recently. Hit rate depends on query distribution.
• →Document embedding cache: Document embeddings are computed offline — no runtime cost.

Monitoring and Drift Detection

• →Retrieval drift: Monitor Context Precision/Recall over time. Drop = embedding drift or corpus change.
• →Answer quality: Monitor faithfulness and user feedback signals (thumbs up/down, follow-up questions).
• →Latency SLAs: P50, P95, P99 latency for each pipeline stage. Alert on P99 spikes.
• →Data freshness: Track when documents were last updated. Stale corpus = stale answers.

MLOps for RAG

• →Index versioning: Treat the vector index like a model artifact. Version it with DVC or a custom registry.
• →Embedding model registry: Never change the embedding model without re-indexing. Track model version as metadata.
• →Eval-driven development: Run RAGAS eval suite on every pipeline change before deploying. Gate on faithfulness > 0.85 and NDCG > 0.7 (example thresholds).
• →A/B testing RAG: Test chunking strategies, K values, or embedding models by routing traffic. Use NDCG and answer correctness as guardrail metrics.

Interview Q&A — Quick Reference

Practice answering each in under 90 seconds.

Quick Reference Cheat Sheet