RAG — Retrieval-Augmented Generation interview questions
RAG — Retrieval-Augmented Generation questions a AI engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
Walk me through a RAG system, from a document being uploaded to a user getting an answer.
The strongest answer: Two pipelines. Ahead of time: parse, chunk, embed and index the documents. At question time: retrieve the closest chunks, put them in the prompt, and have the model answer from them with sources. The weights never change
Why it works: Exactly — ingestion and answering are separate pipelines, and many RAG bugs live in the first one. Nothing is trained: the knowledge arrives in the prompt.
- Two pipelines: ingest, then answer.
- Parse, chunk, embed, index.
- Retrieve, assemble the prompt, generate, cite.
- Nothing is trained; knowledge is in the prompt.
Why do we split documents into chunks before embedding them, instead of making one vector per document?
The strongest answer: One vector for a long document blurs many topics, so specific questions match it poorly, and what we retrieve has to fit the prompt and be citable. Too small, though, and a chunk loses its meaning
Why it works: Exactly — a chunk should be about one thing: small enough to retrieve precisely, large enough to stand on its own. That tension is the whole craft.
- One chunk, one idea.
- Whole-document vectors blur topics.
- Small enough to retrieve and cite.
- Large enough to stand alone.
What is chunk overlap for, and what does too much of it cost?
The strongest answer: It keeps an idea that straddles a boundary whole in at least one chunk. Too much inflates the index and fills the top results with near-duplicates; splitting on headings and paragraphs first reduces the need
Why it works: Right — overlap is insurance against bad boundaries, and it has a premium: storage, embedding cost and duplicate hits that waste the context budget.
- Overlap protects ideas at boundaries.
- Too much means duplicates and cost.
- Split on structure before size.
- Test settings on your own documents.
Users paste the error code “E-4012” into the support assistant. It returns passages about errors in general; the page that documents E-4012 is never among them.
What do you change in retrieval?
The strongest answer: Exact identifiers are where dense embeddings are weakest. I add keyword retrieval — BM25-style — next to the vector search, fuse the two rankings, and put code-and-name queries into the eval set
Why it works: Exactly — hybrid search lets each method cover the other’s blind spot: keywords for exact strings, vectors for paraphrase. The eval queries make sure it stays fixed.
- Dense vectors are weak on exact strings.
- Add keyword search next to vectors.
- Fuse the two rankings.
- Put code queries in the eval set.
A policy document was updated yesterday and the assistant still quotes the old text, with a citation. Where do you look, and what is the real fix?
The strongest answer: Ingestion, not the model: the index still holds the old chunks. The fix is change-driven re-ingestion — detect updates, replace that document’s chunks by document id, re-embed — and deletions have to propagate too
Why it works: Exactly — the model answered faithfully from stale data. Freshness is an ingestion property: updates and deletes must reach the index, keyed by document.
- Stale answers mean a stale index.
- Re-ingest on change, not by hand.
- Replace chunks by document id.
- Propagate deletions as well.
In a multi-turn chat about leave policy, the user asks “and what about for contractors?”. Retrieval returns unrelated passages.
Why does that happen, and what do you do about it?
The strongest answer: The follow-up is not a standalone query; embedded alone it carries no topic. I rewrite it into a self-contained question using the conversation — a cheap model call — retrieve on that, and log the rewrites
Why it works: Exactly — retrieval sees only the query string, not the chat. A rewrite step restores what the user meant, and because it can also get it wrong, you log it and evaluate it.
- Retrieval sees the query, not the chat.
- Rewrite follow-ups into standalone questions.
- Retrieve on the rewrite.
- Log and evaluate the rewrite step.
The assistant gave a wrong answer to a question the documentation clearly covers. How do you find out which stage of the pipeline failed?
The strongest answer: Follow that query’s trace stage by stage: is the passage in the index and parsed properly? Was it retrieved? Did it survive reranking and the context budget? If it reached the prompt and was ignored, it is generation
Why it works: Exactly — a RAG answer can fail at ingestion, retrieval, assembly or generation, and each has a different fix. The trace tells you which; guessing fixes the wrong one.
- Find the stage before the fix.
- Indexed? Retrieved? Kept? Used?
- Read the trace for that query.
- Confirm the fix on the eval set.
The right chunk is nearly always in your top 50 results but rarely in the 5 you send to the model. What do you add, and what does it cost?
The strongest answer: A reranker: retrieve wide and cheap, then score each query–passage pair with a model that reads both together — a cross-encoder, say — and keep the top few. It adds latency per candidate, so I cap the pool
Why it works: Right — recall is fine and precision at the top is the problem, which is exactly what a reranker is for. It is slower per pair than a vector comparison, hence two stages.
- Good recall, poor top-5: rerank.
- Retrieve wide, rerank narrow.
- Rerankers read query and passage together.
- Cap candidates; measure the latency.
A teammate changed the chunk size and swapped the embedding model. Their pull request says the answers “feel better” on the five questions they tried.
You own the RAG system. What do you ask for before this merges?
The strongest answer: A run against a golden set: real questions with their known source passages, including unanswerable ones. Retrieval metrics and answer quality reported separately, before and after — and if that set does not exist, building it comes first
Why it works: Exactly — with labelled source passages you can see whether retrieval found the right text, separately from whether the answer used it. Without a fixed set, every change is an opinion.
- No golden set, no retrieval change.
- Label questions with their source passages.
- Include questions with no answer.
- Report retrieval and answers separately.
Leadership wants one assistant over all internal documents — HR, legal, finance, engineering — for every employee.
What is the first design constraint you raise?
The strongest answer: Permissions. Retrieval must enforce each user’s access at query time: access metadata on every chunk, synced from the source systems, filtered server-side before anything reaches the prompt — including revocations and deletions
Why it works: Exactly — a model cannot keep a secret it has been handed. What a user may not open must never be retrieved for them, and access changes in the source have to reach the index.
- A model cannot keep a secret.
- Enforce access at retrieval, server-side.
- Mirror source permissions on every chunk.
- Propagate revocations and deletions.
Since you added “answer only from the retrieved context, otherwise say you don’t know”, the assistant refuses about a third of the questions the docs do cover. Sales wants the rule removed before a customer demo.
What do you do?
The strongest answer: Keep the rule and split the refusals by cause: where retrieval missed, fix retrieval; where the context was good, fix the prompt — allow partial answers that say what is covered. Track false refusals and unsupported answers together
Why it works: Exactly — over-refusal and unsupported answers are two error rates on one dial, and you manage both. A false refusal has a cause you can find — a missed passage or an over-strict prompt — without touching the grounding rule.
- Two error rates: false refusals, unsupported answers.
- Split refusals by cause.
- Fix retrieval or prompt; keep grounding.
- Partial answers should say what is covered.