AI Evals interview questions
AI Evals questions a AI engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
Why can’t you test an LLM feature with ordinary exact-match unit tests, and what do you do instead?
The strongest answer: The output varies between runs and many wordings are correct, so exact match breaks; I run a set of cases, grade properties — format, facts, criteria — and track the pass rate
Why it works: Right — you test behaviour over a set, not one string. A pass rate over graded cases is the LLM equivalent of a test suite.
- Grade properties over a set, not one string.
- Expect variation between runs.
- Check format, facts and criteria.
- Track the pass rate over time.
What is a golden dataset for an LLM feature, and where do its cases come from?
The strongest answer: A curated, versioned set of inputs with reviewed expected outcomes, drawn mostly from real user traffic and past failures, with edge cases included — not just happy-path examples I made up
Why it works: Exactly — it has to look like production and remember every failure you have fixed, or a good score tells you nothing.
- Production-like cases with reviewed answers.
- Sample real traffic.
- Add every failure you fix.
- Version it; review the labels.
Your feature extracts supplier, date and total from invoices into JSON. A teammate proposes grading every eval case with an LLM judge.
What do you suggest instead?
The strongest answer: Code checks wherever the answer is checkable — valid schema, exact field match, numeric equality on the total — because they are cheap, fast and repeatable; keep model grading for open-ended qualities
Why it works: Exactly — use the most deterministic grader that can decide the case. A judge adds cost and noise where an equality check is already certain.
- Use the most deterministic grader that works.
- Schema and exact match in code.
- Judges only for open-ended quality.
- Similarity is not correctness.
You compare an old and a new prompt with a pairwise LLM judge. The new prompt wins almost every pair. Its answers are always shown second, and they are longer.
What do you check before you believe the result?
The strongest answer: Position and verbosity bias: run every pair in both orders and count only verdicts that survive the swap, check whether length alone is winning, and compare a sample with human preferences
Why it works: Exactly — judges can favour a position or the longer answer. Swap the order, control for length, and calibrate against people.
- Pairwise judges have position and length bias.
- Run both orders; keep consistent verdicts.
- Check whether longer always wins.
- Calibrate on a human-labelled sample.
How do you make evals part of CI, so a prompt or retrieval change cannot quietly make things worse?
The strongest answer: A fast fixed subset runs on every change to prompts, model settings or retrieval code and is compared with the stored baseline; a real drop blocks the merge, and the full suite runs before release
Why it works: Exactly — a quick gate on every change, a baseline to compare with, and a threshold that allows for run-to-run noise. The slow suite guards the release.
- A prompt change gets a gate like code.
- Fast subset on every change.
- Compare with a stored baseline.
- Allow for noise; full suite before release.
Your assistant can call six tools. How do you evaluate whether it uses them correctly?
The strongest answer: Cases with an expected call: right tool — or none when none is needed — and valid, correct arguments, all checked in code, plus cases where it should ask for a missing detail rather than guess
Why it works: Exactly — selection, arguments and restraint are three separate failure modes, and all three are checkable without a judge.
- Grade the call, not only the answer.
- Right tool, or no tool.
- Arguments valid and correct.
- Include cases where it should ask.
Answers got worse after you changed the chunking in a RAG assistant. How do you evaluate the retriever on its own?
The strongest answer: A labelled set of queries with the source passages that answer them; measure recall at k and the rank of the first relevant hit for old and new chunking on the same queries, before any generation
Why it works: Exactly — relevance labels tied to the source text survive a re-chunk, and recall at k tells you whether the answer was even available to the generator.
- Measure retrieval before generation.
- Label queries with their relevant sources.
- Compare recall at k and first-hit rank.
- Same queries, old versus new.
Your offline eval score rose for three releases in a row. Over the same period, escalations from the assistant to human support went up.
What is going on, and what do you do?
The strongest answer: The offline set or its grader has drifted from production. I sample escalated sessions, label them, check the grader against those labels, fold them into the set, and keep an online signal beside the offline score
Why it works: Exactly — when the two disagree, the offline eval is the instrument under suspicion. Rebuild it from the sessions where users actually failed.
- When offline and online disagree, audit offline.
- Read and label the failing sessions.
- Validate the grader against humans.
- Refresh the set; keep both signals.
How do you evaluate a multi-step agent, where two correct runs can take completely different paths?
The strongest answer: Grade the outcome by checking the end state in a sandbox, not the agent’s claim; add trajectory checks — forbidden actions, wasted steps, cost — and run each task several times, because one pass proves little
Why it works: Exactly — verify what changed in the world, constrain the path only where the path matters, and treat success as a rate across trials.
- Grade the end state, not the story.
- Verify outcomes in a sandbox.
- Check forbidden actions, steps and cost.
- Repeat trials; report a success rate.
An executive asks for “our hallucination rate” as one number for a board deck about your document-grounded assistant.
What do you give them?
The strongest answer: A defined measurement: on a stated sample of production-like questions, the share of answers with a claim the sources do not support, labelled by humans or a judge validated against them, with sample size and limits
Why it works: Exactly — a rate means nothing without its definition, its sample and its labeller. Stating the limits is what makes the number defensible.
- A rate needs a definition and a sample.
- Define unsupported claim against the sources.
- Validate the labeller against humans.
- Report sample size and limits.