Practical AI System Design interview questions
Practical AI System Design questions a AI engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
You are asked to design a simple chatbot for a company’s website. Walk me through the building blocks.
The strongest answer: First who it is for and what it must never say. Then: chat UI, a backend that holds the key and system prompt, a conversation store, the model call, logging and usage limits, and test questions
Why it works: Exactly — requirements first, then a short list of boring components, and a way to check that it works. That is a design, even for a small bot.
- Requirements before components.
- UI, backend, store, model call.
- The key stays on the server.
- Limits, logs and test questions.
Design a bot that answers employees’ questions from the company’s policy documents. What are the parts?
The strongest answer: Two paths: ingestion parses, chunks, embeds and indexes the documents; the query path retrieves relevant chunks and has the model answer only from them, with citations, and say it does not know when nothing fits
Why it works: Exactly — retrieval supplies the facts, the instruction keeps the model to them, and citations let an employee check. The “I don’t know” path matters as much as the answer path.
- Ingest once, retrieve per question.
- Parse, chunk, embed, index.
- Answer only from the retrieved text.
- Cite sources; allow “I don’t know”.
A finance team wants a tool that reads supplier contracts (PDFs, some of them scanned) and pulls the parties, dates, payment terms and renewal clauses into their system.
How do you design it?
The strongest answer: Ask which fields matter and what an error costs. Then parse (OCR for scans), extract into a strict schema with a source quote per field, validate, route doubtful fields to a reviewer, and measure per-field accuracy on labeled contracts
Why it works: Exactly — this is an extraction pipeline, not a chatbot. The source quote makes each field checkable, and per-field accuracy shows where review is still needed.
- An extraction pipeline, not a chatbot.
- Strict schema, source quote per field.
- Validate; send doubtful fields to review.
- Measure accuracy per field.
IT wants access requests handled automatically: read the free-text request, look up the employee’s role in the HR system, pick the matching standard access package, and create the accounts.
Do you build an agent that chooses its own steps, or something else? Design it.
The strongest answer: The steps are known, so a fixed workflow: the model turns the request into structured fields, code looks up the role and maps the package, and account creation is a validated call, with approval for anything non-standard
Why it works: Exactly — the model handles the messy language and plain code handles the rules. Autonomy is a cost worth paying only when the steps cannot be written down.
- Known steps: a workflow, not an agent.
- Model for language, code for rules.
- Validate every tool call’s arguments.
- Approval for anything non-standard.
Design a customer-support agent for a subscription product: it answers questions and handles simple account actions. Where do you start, and what are the parts?
The strongest answer: Scope and risk first: which intents it owns, what it must never do, when it hands off. Then grounded help-center answers, a few narrow tools scoped to the signed-in customer, confirmation before changes, and evals from real tickets
Why it works: Exactly — the design starts from what failure costs. Tools that act only as the signed-in customer mean a confused or manipulated agent cannot touch anyone else’s account.
- Scope and failure cost come first.
- Ground answers in the help center.
- Narrow tools, scoped to the customer.
- Confirm changes; hand off with context.
Design an AI interview coach: the candidate answers practice questions aloud or in text and gets feedback. What drives the design?
The strongest answer: Two budgets: the conversation must feel live, so turns are fast and streamed; grading runs afterwards, against a written rubric, and quotes the candidate’s own words. No hiring predictions, and the grader is checked against human-scored answers
Why it works: Exactly — latency where a person is waiting, quality where they are not. A rubric keeps the feedback consistent, and comparing the grader with human scores tells you whether to trust it.
- Live turns fast; grading can wait.
- Grade against a written rubric.
- Quote the answer; invent no numbers.
- Check the grader against human scores.
A team proposes rebuilding its research-report generator as five cooperating agents: planner, searcher, analyst, writer and critic. Today it is one prompt with a search tool, and the reports are sometimes shallow.
You are reviewing the design. What do you ask, and when is multi-agent the right call?
The strongest answer: First, what exactly fails today, measured on real reports? Try cheaper fixes: better search, a plan-then-write chain, a review pass. Separate agents pay off when subtasks are independent, parallel, or need their own context or permissions
Why it works: Exactly — diagnose before you redesign. Every agent adds tokens, latency, handoff errors and one more trajectory to evaluate, so each must buy something a simpler chain cannot.
- Diagnose the failure before redesigning.
- Try the simplest fix first.
- Split only for independence or isolation.
- Each agent must earn its cost.
Your assistant works well for a few thousand users, and a partnership will bring ten times the traffic in three months. What breaks first, and what do you do about it?
The strongest answer: Usually provider quota and the bill, before our servers. Secure capacity early, add per-tenant limits and queues, cut tokens per request, cache, route easy work to cheaper models, and load-test with realistic prompts
Why it works: Exactly — in an LLM application the scarce resources are tokens per minute and money per request. Stateless servers scale easily; the quota and the unit cost do not.
- Tokens and quota are the bottleneck.
- Secure provider capacity early.
- Spend fewer tokens per request.
- Load-test with realistic prompts.
You are designing an assistant that reads a user’s email and calendar, browses web pages, and can send messages on the user’s behalf.
How do you keep it safe, given that it reads content written by strangers?
The strongest answer: Treat every email and page as possibly hostile instructions that no prompt reliably blocks. So bound the damage: least-privilege tokens, user approval before anything is sent, restricted outbound channels, audit logs, and injection cases in the evals
Why it works: Exactly — you cannot make the model immune, so you design for the day it is fooled. Private data, untrusted content and an outbound channel are the combination you break up.
- Design for the day it is fooled.
- Least privilege on every tool.
- Approval before anything leaves.
- Log actions; test with injections.