AI / LLM Fundamentals interview questions
AI / LLM Fundamentals questions a AI engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
What is a token, and why does an engineer building on an LLM care?
The strongest answer: The unit the model reads and writes — a piece of a word, not a word. Limits, cost and speed are all counted in tokens, and the count varies with language and content
Why it works: Exactly — context limits, billing and latency are all measured in tokens, and code, long numbers or non-English text often take more tokens than the word count suggests.
- Tokens are the model’s unit of text.
- Limits, cost and latency count tokens.
- Words are only a rough estimate.
- Count with the tokenizer when it matters.
What does the temperature setting do, and how do you choose it for a task?
The strongest answer: It controls how random each next-token choice is: low keeps to the most likely tokens, high spreads out. Low for extraction and classification, higher for brainstorming and varied wording
Why it works: Right — it reshapes the probability distribution the next token is sampled from, so you pick it by how much variation the task can tolerate.
- Temperature sets sampling randomness.
- Low for extraction and classification.
- Higher for ideas and varied wording.
- It does not fix hallucination.
Why does an LLM sometimes state something false with complete confidence?
The strongest answer: It generates the most plausible continuation, not a looked-up fact, and nothing in that process checks truth or signals doubt — so a fluent wrong answer reads exactly like a right one
Why it works: Exactly — fluency and truth are separate. The model produces likely text, which is why you add sources, checks and a way to say “I do not know”.
- Plausible text is not checked fact.
- A confident tone tells you nothing.
- Supply sources for facts that matter.
- Allow and test “I do not know”.
Your assistant confidently describes an old version of a library that had a major release last month. What is going on, and what do you change?
The strongest answer: Its knowledge stops at a training cutoff and it cannot tell what has changed since. I put today’s date and current sources in the context, and have it flag what those sources do not cover
Why it works: Exactly — the fix is in the context: current date, current sources, and permission to say “I cannot confirm that”.
- Knowledge ends at the training cutoff.
- Give it the date and fresh sources.
- Let it say what it cannot confirm.
- A disclaimer is not a fix.
Temperature is zero, yet a test that asserts the exact text of the model’s reply fails now and then. Why, and how should the test be written?
The strongest answer: Zero temperature narrows variation, but hosted inference is not bit-for-bit repeatable — parallel floating-point arithmetic, batching and provider updates all shift outputs. I assert on structure and meaning, and pin the model version
Why it works: Right — treat exact-string equality as the bug. Check parsed fields, required facts or a graded rubric, and record which model version produced the result.
- Temperature zero is not a determinism guarantee.
- Assert on structure and meaning.
- Pin and record the model version.
- Never retry until green.
Why does a long answer usually take much longer than a long prompt, and what does that change in your design?
The strongest answer: The prompt is processed in parallel in one pass, but output is generated one token at a time, each needing its own pass. So I cap and shorten outputs, and stream them to the user
Why it works: Exactly — prefill is parallel, decoding is sequential. Output length is your main latency lever: ask for less, structure it tightly, and stream what remains.
- Input is parallel; output is sequential.
- Output length drives total latency.
- Ask for shorter, tighter output.
- Stream for time to first token.
When do you reach for a reasoning model, and when is it the wrong choice?
The strongest answer: For multi-step work — planning, hard code, maths — where thinking before answering buys accuracy worth the extra latency and tokens. Not for simple extraction or latency-bound chat. An eval on my own task decides
Why it works: Exactly — it spends extra tokens working through the problem first. That pays on multi-step tasks and is waste on lookups; your own eval settles which is which.
- Reasoning models trade time for accuracy.
- Use for multi-step, high-value tasks.
- Skip for simple or instant tasks.
- Let your own eval decide.
Every proposal that reaches you now says “use an LLM”: churn prediction from a customer table, demand forecasting, fraud scoring, and a support-ticket summariser.
How do you sort these, and what do you tell the teams?
The strongest answer: Match technique to data: tabular prediction and forecasting go to classical ML — cheaper, faster, measurable, explainable. Language work like the summariser suits an LLM. Each proposal brings a simple baseline first
Why it works: Exactly — generative AI is one branch of machine learning, not its replacement. A gradient-boosted model on a customer table is still the honest baseline to beat.
- Pick the technique the data calls for.
- Tabular and forecasting: classical ML first.
- Language tasks: LLM candidates.
- Demand a baseline before any build.
Your assistant drafts replies that a human reviews before sending. Leadership asks when hallucinations will be “fixed” so the review step can be removed.
What do you tell them?
The strongest answer: Never to zero — it comes with how generation works. We remove review by risk tier, where our measured error rate on our own traffic is acceptable and mistakes are cheap to undo, and keep it elsewhere
Why it works: Exactly — reframe from “when is it fixed” to “where is the residual error acceptable”. That is a measured, per-tier decision you can defend after an incident.
- Hallucination is reduced, never eliminated.
- Measure error on your own traffic.
- Remove review by risk tier.
- Keep humans where mistakes are costly.