AI Observability interview questions
AI Observability questions a AI engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
What do you record for each LLM call so that you can debug it later?
The strongest answer: The prompt as sent, or its version and variables, the model and settings, the response, token counts, latency, any error, and an id linking it to the user’s request — with sensitive content handled deliberately
Why it works: Right — you cannot debug an answer without seeing exactly what the model was given, by which version, at what cost and in what time.
- Record what the model actually saw.
- Prompt, model, settings, response.
- Tokens, latency, errors.
- One id back to the user request.
To debug a chatbot, a teammate adds a line that writes every full prompt and response into the general application log.
What concerns you, and what do you suggest?
The strongest answer: Prompts carry whatever users type — names, health details, pasted secrets — and that log is widely readable and kept a long time; I would mask sensitive fields, restrict access and set a retention period
Why it works: Exactly — a prompt log is a store of user data. Treat it like one: minimise, restrict, expire.
- Logged prompts are user data.
- Mask sensitive fields before storing.
- Restrict who can read traces.
- Set a retention period.
A user reports that your agent gave a confidently wrong answer yesterday. All you stored was the final answer.
What is missing, and what would you add?
The strongest answer: A trace of the whole run: each model call and tool call as a step under one trace id, with inputs, outputs, errors and timings — so I can see which step went wrong
Why it works: Exactly — in a multi-step run the wrong answer is usually made several steps before the last one. The trace shows where.
- Trace the run, not just the answer.
- One trace id per user request.
- A step per model and tool call.
- Inputs, outputs, errors, timings.
Your LLM bill doubled this month while the number of requests stayed flat. How do you find out why?
The strongest answer: Break token usage down per request by feature, prompt version and model, input against output: with flat traffic something grew — longer history, more retrieved context, retries, or an agent taking more steps
Why it works: Exactly — cost follows tokens, so with flat requests you look for tokens per request growing, and the per-feature breakdown says where.
- Flat traffic, bigger bill: tokens per request.
- Record tokens on every call.
- Slice by feature, version and model.
- Look for growing context, retries, loops.
Complaints about answer quality jumped this week, and your team deployed nothing. How does your observability help, and what had to be in place beforehand?
The strongest answer: Every trace carries the prompt version, the exact model version rather than an alias, and the retrieval config, so I can slice quality by each and see what moved: the model, the index or the traffic
Why it works: Exactly — “we deployed nothing” is rarely “nothing changed”. Version stamps on every trace turn a mystery into a group-by.
- Stamp every trace with every version.
- Prompt version on each call.
- Exact model version, not the alias.
- Slice quality by version and date.
You have a few hundred traces that users marked as bad. How do you turn them into improvements?
The strongest answer: Read a sample end to end, write down what went wrong in each, group the notes into failure categories and count them; fix the biggest category first and add its cases to the eval set
Why it works: Exactly — categories with counts turn a pile of complaints into a ranked backlog, and the cases become regression tests for the fix.
- Read, categorise, count, then fix.
- Read real traces end to end.
- Group failures; count each group.
- Turn fixed cases into eval cases.
Error-rate and latency alerts are already in place for your LLM feature. What else do you monitor in production, and why?
The strongest answer: Quality fails without throwing errors, so I track proxies — refusal, fallback and validation-failure rates, tool errors, retries and thumbs-down, cost per request — and score a sample of live traffic, alerting on shifts by version
Why it works: Exactly — a confidently wrong answer returns 200 in good time. Proxies and sampled scoring are how a quality regression pages someone.
- Bad answers do not throw errors.
- Track refusals, fallbacks, validation failures.
- Watch user signals and cost per request.
- Score a live sample; alert on shifts.