RAG Interview Questions: The Retrieval Ladder Behind the Hype

RAG is the topic where LLM interviews find out whether you have built a system or wired a demo. The demo is easy: embed some documents, retrieve the top few by similarity, paste them into the prompt, and the model answers. That version works in a notebook and falls apart in production, and a good interviewer knows exactly where. So the real rag interview questions escalate past the pipeline everyone can recite into the parts that decide whether the system actually answers correctly: how you chunk, how you fix retrieval, how you evaluate the thing, and why it still hallucinates after all of that.

This guide walks the ladder the way an interviewer climbs it. For each rung: the question as it gets asked, what it is really probing, the answer at the depth that lands, and the trap that ends the conversation early. It is the retrieval companion to attention mechanism interview questions, and it turns on one load-bearing idea you should be ready to defend at every rung: RAG is a retrieval problem wearing a generation costume.

Rung 1: "What problem does RAG actually solve, and why not just fine-tune or use a long context?"

What it probes: whether you can position RAG against its alternatives, which is the reasoning move that separates a decision-maker from someone reciting a definition.

The answer that lands names what RAG is for and what it competes with. RAG grounds a model's output in retrieved documents so it can answer over a corpus it was not trained on, cite sources, and stay current as that corpus changes, without touching the weights. Its real competitors are fine-tuning and long-context stuffing, and the honest framing is that they solve different problems. Fine-tuning changes behavior and style and teaches durable skills, but it is a poor way to inject knowledge: the facts get baked in, go stale, and cannot be cited or updated without retraining. Long context lets you paste more in, but it is expensive per query, degrades on recall in the middle of very long inputs, and does not scale to a corpus larger than the window. RAG is the right tool when the knowledge is large, changing, and needs attribution.

The trap: "RAG is how you give an LLM your own data," full stop. True and shallow. The interviewer wants the comparison: why not fine-tune (knowledge injection is fine-tuning's weakest use, and it cannot cite), why not just use a 1M-token window (cost per query and mid-context recall). A candidate who reasons about the alternatives has shown the judgment the rest of the ladder builds on. The deeper treatment of where retrieval fits among these is in RAG in 2026: context engines and GraphRAG.

Rung 2: "Walk me through a basic RAG pipeline"

What it probes: whether you can name the stages precisely, so the interviewer knows which one to attack next.

The clean answer has two phases. Offline (indexing): split documents into chunks, embed each chunk with an embedding model into a dense vector, and store those vectors in an index. Online (query time): embed the user's query with the same model, retrieve the top-k chunks by vector similarity (cosine or dot product over an approximate-nearest-neighbor index), assemble them into the prompt, and let the model generate an answer grounded in them. Naming the stages cleanly matters because every later rung is an attack on one of them: chunking on the split step, reranking on the retrieve step, eval on the whole chain.

The trap: blurring indexing and query time, or forgetting that the same embedding model must encode both documents and queries into the same space. The subtle tell an interviewer listens for is whether you know retrieval is approximate: production vector search uses ANN indexes (HNSW and friends) that trade a little recall for speed, so "top-k by similarity" is already an approximation with a recall knob, not an exact lookup. Mentioning that unprompted signals you have run this at scale, not just in a tutorial.

Rung 3: "How do you decide how to chunk?"

What it probes: whether you understand that chunking is a lossy, consequential decision, or you reach for "500 tokens with 50 overlap" as a reflex.

The strong answer frames chunking as a tension. Chunks that are too small lose the context needed to be understood on their own (a pronoun whose referent is in the previous chunk, a number whose units are in the heading). Chunks that are too large dilute the embedding: a single vector is asked to represent many ideas, so it matches queries weakly and drags in irrelevant text when retrieved. The right unit is usually semantic, not a fixed token count: split on document structure (sections, paragraphs, headings) so each chunk is one coherent idea, and carry metadata (source, section title, surrounding context) alongside the vector. Techniques like retrieving small chunks but feeding the model their larger parent section, or prepending a short LLM-generated summary of the document to each chunk, exist precisely to break the small-versus-large tension.

The trap: "500 tokens with 50 overlap" as a universal answer. Fixed-size chunking is a default, not a decision, and reciting it signals you have never watched it fail. The interviewer wants to hear that chunk boundaries determine what can be retrieved at all: if the answer to a question is split across a boundary, no retriever can assemble it, so chunking is upstream of every retrieval-quality problem later in the ladder. A candidate who ties chunking to retrievability has understood why it is the quiet root cause of a lot of RAG failures.

Rung 4: "Your retrieval is not good enough. What do you do?"

What it probes: whether you know that retrieval quality is the bottleneck of the whole system, and can name the levers in the order that matters.

This is the rung that decides senior RAG interviews, and the load-bearing sentence is: the generator can only be as good as what you retrieved, so a wrong or missing chunk is an unrecoverable error no amount of prompting fixes. From there the levers follow. First, the embedding model itself: retrieval quality varies a lot across models, and the honest way to choose is by the retrieval sub-score on a benchmark like MTEB, not the headline average (a model with a high overall score can have a mediocre retrieval score, because the headline blends in clustering and classification tasks you do not care about). Current strong retrievers (Qwen3-Embedding, Voyage, Cohere Embed, and the decoder-LLM-backbone models topping MTEB retrieval) are worth benchmarking on your own data rather than trusting a leaderboard rank.

Second, hybrid search: combine dense vector retrieval with sparse keyword retrieval (BM25), because embeddings miss exact matches (product codes, names, rare terms) that lexical search nails, and fusing the two rankings catches what either misses alone. Third, and highest-leverage, reranking: retrieve a generous candidate set (say top-50) with the cheap bi-encoder, then rerank it with a cross-encoder that scores each query-document pair jointly. The cross-encoder is far more accurate than embedding similarity because it attends to query and document together, and running it only on the shortlist keeps it affordable. Hybrid retrieval plus a reranker is the production baseline for most systems.

The trap: jumping to "make the prompt better" or "use a bigger model." Both are downstream of the real failure. If the right chunk is not in the retrieved set, the generator has nothing to work with, and every fix that touches the prompt or the model is treating a retrieval failure at the generation layer. The senior answer fixes retrieval first, in the order embedding model, hybrid search, reranker, and treats the LLM as the last thing to touch.

Rung 5: "How do you evaluate a RAG system?"

What it probes: whether you can separate a retrieval failure from a generation failure, because they live in different places and are fixed by different people.

The strong answer evaluates the two stages separately. Retrieval gets information-retrieval metrics: recall@k (did the relevant chunk make it into the top-k at all, the single most important RAG metric because it is the ceiling on everything downstream), plus precision and nDCG for ranking quality. Generation gets grounding metrics: faithfulness (does the answer follow from the retrieved context, or did the model invent beyond it), answer relevance (does it address the question), and context relevance (was the retrieved context actually on-topic). Only by measuring both can you attribute a bad answer: low recall means fix retrieval, high recall but low faithfulness means fix the generation step or the prompt. Building a small labeled eval set of question-and-expected-source pairs is what turns RAG from vibes into engineering.

The trap: evaluating only the final answer, usually with an LLM-as-judge score on the output. That single number cannot tell you whether the system retrieved the wrong context or retrieved the right context and then ignored it, and those two failures are fixed in completely different places. A judge score on the answer alone is half a story, and treating it as the whole story is the same eval mistake the field keeps making with leaderboard numbers. The broader case for staged, honest evaluation is in the LLM evaluation crisis.

Rung 6: "It retrieves the right documents and still hallucinates. Why?"

What it probes: whether you know the failure modes that survive good retrieval, which is where candidates who aced rung four often assume the problem is solved.

The strong answer enumerates the failure modes past retrieval. The model can ignore the context and answer from its parametric memory, especially when the retrieved passage contradicts what it "knows." It can suffer lost-in-the-middle: when many chunks are stuffed in, information in the middle of a long context gets attended to less than the ends, so the relevant chunk is present but effectively unread. The context can be internally contradictory (two retrieved chunks disagree, and the model picks one arbitrarily or blends them). Or the question needs information from multiple documents that no single chunk contains, so even perfect single-hop retrieval cannot assemble the answer. Naming these shows you understand that "retrieved correctly" and "answered correctly" are separate guarantees.

The trap: blaming the model and reaching for a bigger one. Most of these are addressable in the system, not the weights: rerank and trim so the best chunks sit where attention is strongest rather than buried in the middle, cap the number of chunks so you do not dilute, instruct the model to answer only from context and to say when the context is insufficient, and detect multi-hop questions that single-hop retrieval cannot serve (the on-ramp to the last rung). A candidate who treats hallucination-after-retrieval as a set of nameable, fixable failure modes has shown the operational maturity the rung is testing.

Rung 7: "When do you need agentic RAG or GraphRAG?"

What it probes: whether you know the boundary of single-shot retrieval, and can name what lies past it without treating every new acronym as an upgrade.

The strong answer draws the line at the query type. Single-shot top-k retrieval works for direct factual lookups where the answer lives in one place. It breaks on two shapes. Multi-hop and relational questions ("which projects did the people who reported to X also work on") need information stitched across entities, which is where GraphRAG helps: it retrieves over a knowledge graph of entities and relationships, so structure, not just similarity, drives what comes back. Open-ended or decomposable questions need agentic RAG: an agent plans, issues multiple retrieval queries, reads intermediate results, and reformulates, turning retrieval from one shot into a loop. The honest framing is that these are not universally better, they are answers to specific query shapes, and hybrid vector-plus-BM25-plus-reranker with selective graph enrichment for known multi-hop patterns is the right default before reaching for a full knowledge-graph build.

The trap: presenting GraphRAG or agentic RAG as the next tier everyone should adopt. They add real cost (graph construction and maintenance, or many more LLM calls per query) and pay off only on the query shapes that need them. The senior answer names the query type each one serves and defaults to the simpler stack until the workload proves it needs more, which is the same "match the tool to the problem, not the hype" discipline the whole ladder rewards. Where retrieval ends and persistent state begins is the subject of agent memory beyond RAG, and the cost side of long inputs is in effective context length.

How to actually prepare (the operational part)

Reading answers is not preparation. RAG questions reward people who can attribute failures to a stage, so rehearse the pipeline as a chain of failure points, not a list of components. A protocol that works:

1. Be able to say the one load-bearing sentence cold: retrieval quality is the ceiling. The generator cannot answer from what it did not retrieve, so a missing chunk is unrecoverable. Every fix has to be located at the right stage, and this sentence is why.
2. Draw the pipeline and mark the failure at each stage. Chunking (answer split across a boundary), embedding (wrong model, mediocre retrieval sub-score), retrieval (right chunk outside top-k), generation (context ignored, lost-in-the-middle). If you can point at the stage, you can propose the fix.
3. Practice the retrieval-fix order out loud: embedding model, hybrid search, reranker, then prompt. Reaching for the prompt or a bigger model first is the tell of someone who treats RAG as a generation problem. The order signals you know it is a retrieval problem.
4. Rehearse the two-stage eval. Recall@k for retrieval, faithfulness and answer relevance for generation. Be ready to say why a single answer-level score cannot separate a retrieval failure from a generation failure.
5. Prepare one honest "where it breaks." Multi-hop questions single-shot retrieval cannot serve, hallucination that survives correct retrieval, chunk boundaries that make an answer unretrievable. Leading with a failure boundary beats claiming RAG solves knowledge cleanly.

The unifying thread across all seven rungs: interviewers are not checking whether you can name a vector database. They are checking whether you understand that RAG is a retrieval system with a language model on the end, so you can locate a failure at the stage that caused it and fix it there. Every rung has a demo-level answer that stalls and a systems-level answer that opens the next door. Prepare for the door.

Key Takeaways

1. RAG competes with fine-tuning and long context, and it is the right tool for large, changing, citable knowledge. Fine-tuning is a poor way to inject facts (they go stale and cannot be cited); long context is expensive per query and degrades on mid-context recall. Position RAG against these rather than defining it in isolation.
2. Name the pipeline precisely, and know retrieval is approximate. Offline: chunk, embed, index. Online: embed query, retrieve top-k by ANN similarity, assemble, generate. Production vector search trades recall for speed, so "top-k" is an approximation with a recall knob, not an exact lookup.
3. Chunking is a lossy decision upstream of every retrieval problem. Too small loses context, too large dilutes the embedding. Split on semantic structure, not a fixed token count, and remember that an answer split across a chunk boundary is unretrievable by any downstream technique.
4. Retrieval quality is the ceiling on the whole system. Fix it in order: embedding model (choose by retrieval sub-score, not headline MTEB), hybrid dense-plus-BM25 search, then a cross-encoder reranker over a generous candidate set. Touching the prompt or the model first treats a retrieval failure at the wrong layer.
5. Evaluate retrieval and generation separately. Recall@k is the most important RAG metric because it caps everything downstream; faithfulness and answer relevance grade the generation step. A single answer-level judge score cannot distinguish "retrieved the wrong context" from "retrieved the right context and ignored it."
6. Hallucination survives good retrieval, and the fixes are systemic. The model can ignore context, lose the middle of a long prompt, get contradictory chunks, or face a multi-hop question single-shot retrieval cannot serve. Rerank and trim, cap chunk count, instruct answer-from-context-only, and detect multi-hop, before blaming the weights.
7. GraphRAG and agentic RAG answer specific query shapes, not a universal upgrade. GraphRAG serves multi-hop relational queries via a knowledge graph; agentic RAG serves decomposable questions via a retrieval loop. Both add real cost, so default to hybrid-plus-reranker with selective graph enrichment until the workload proves it needs more.