AI engineer interview questions

What is a token, and why does an engineer building on an LLM care?

The strongest answer: The unit the model reads and writes — a piece of a word, not a word. Limits, cost and speed are all counted in tokens, and the count varies with language and content

Why it works: Exactly — context limits, billing and latency are all measured in tokens, and code, long numbers or non-English text often take more tokens than the word count suggests.

• Tokens are the model’s unit of text.
• Limits, cost and latency count tokens.
• Words are only a rough estimate.
• Count with the tokenizer when it matters.

What is the difference between conversation memory and user memory in an assistant?

The strongest answer: Conversation memory is this session’s turns, resent to the model as context; user memory is facts the app stores across sessions and retrieves when relevant. The application builds both — the model keeps nothing

Why it works: Right — the model is stateless between calls. Short-term memory is what you resend; long-term memory is a store you design, write to and retrieve from.

• The app builds memory, not the model.
• Conversation memory: this session, resent as context.
• User memory: stored facts across sessions.
• Select what is kept; retrieve what matters.

What makes something an agent rather than a fixed LLM workflow?

The strongest answer: A workflow has its steps fixed in my code, with the model filling each one in; an agent lets the model decide the next step itself, looping with tools until the task is done

Why it works: Exactly — the difference is who owns the control flow. And when the steps are known in advance, the workflow is the cheaper, more predictable choice.

• Who decides the next step?
• Workflow: code owns the sequence.
• Agent: the model chooses, in a loop.
• Known steps? Prefer the workflow.

def add(item, bag=[]): bag.append(item); return bag — you call add(1) then add(2). What comes back the second time?

The strongest answer: [1, 2] — the default list is created once when the function is defined, so every call without a bag shares it; use bag=None and build a new list inside

Why it works: Exactly. One default object, shared for the life of the function. None plus a fresh list inside is the fix.

• Default arguments are evaluated once, at definition.
• A mutable default is shared by every call.
• Use None and create the value inside the function.
• The bug is silent, not an error.

An interviewer says: “Design a URL shortener.” What do you do first?

The strongest answer: Ask what it must do and at what scale — reads vs writes, expiry, custom aliases, analytics — and state the assumptions you will design against before drawing anything

Why it works: Right. Requirements and numbers first; the boxes follow from them.

• Clarify functional and non-functional requirements first.
• State assumed scale in numbers.
• Only then draw.
• Check assumptions with the interviewer.

Walk me through a RAG system, from a document being uploaded to a user getting an answer.

The strongest answer: Two pipelines. Ahead of time: parse, chunk, embed and index the documents. At question time: retrieve the closest chunks, put them in the prompt, and have the model answer from them with sources. The weights never change

Why it works: Exactly — ingestion and answering are separate pipelines, and many RAG bugs live in the first one. Nothing is trained: the knowledge arrives in the prompt.

• Two pipelines: ingest, then answer.
• Parse, chunk, embed, index.
• Retrieve, assemble the prompt, generate, cite.
• Nothing is trained; knowledge is in the prompt.

What is a guardrail in an LLM application, and where does it sit?

The strongest answer: A check outside the model that enforces policy the prompt cannot guarantee — on the input before the call and on the output before it reaches a user or a tool — written as code or a separate classifier

Why it works: Exactly — the prompt asks, the guardrail checks. Because it runs outside the model, it still works on the day the model ignores its instructions.

• The prompt asks; the guardrail checks.
• Check input before the model call.
• Check output before users or tools.
• Keep the policy in code you own.

When would you choose a tuple over a list?

The strongest answer: When the collection is fixed and its position means something — a coordinate, a record, a dict key — since tuples are immutable and therefore hashable

Why it works: Right. Immutability is the point, and hashability is what it buys you.

• Tuple = fixed, positional, hashable.
• List = a sequence you intend to change.
• Only a tuple can key a dict or join a set.
• Choose on meaning, not on length.

What does a load balancer do, and what does it not do?

The strongest answer: It spreads incoming requests across several identical servers and stops sending to ones that fail health checks; it does not make one slow server faster or fix a database bottleneck behind them

Why it works: Right. Distribution and failover, nothing more.

• Distributes requests across servers.
• Removes unhealthy servers.
• Not a cache, not capacity.
• Bottlenecks behind it remain.

A teammate wants to call an LLM to work out the sales tax on each invoice. What do you say?

The strongest answer: That is a rule with one right answer, so plain code does it exactly, instantly, cheaply and testably. A model earns its place on fuzzy input — language, judgment, variation — where rules cannot be written down

Why it works: Exactly — deterministic problems want deterministic software. The model can still help next to it, for example reading a messy invoice before the code does the sums.

• Rules in code, fuzziness in models.
• One right answer? Write code.
• Messy language or judgment? Consider a model.
• Never leave money math unchecked.

What is the difference between == and is?

The strongest answer: == asks whether two values are equal; is asks whether they are the very same object — use is only for None, True, False and sentinels

Why it works: Exactly. Identity versus equality, and is belongs to singletons.

• == compares values, is compares identity.
• Use is None, not == None.
• Equal objects need not be the same object.
• Caching makes is look right by accident.

What is the difference between scaling up and scaling out?

The strongest answer: Up means a bigger machine — simple, but there is a ceiling and one box to lose; out means more machines behind a balancer, which needs the service to be stateless or to share state elsewhere

Why it works: Right. Up is easy until the ceiling; out needs stateless services.

• Up: bigger box, ceiling, single failure.
• Out: more boxes, needs statelessness.
• Choose by state and ceiling.
• Neither is always right.

How do you decide what gets written to a user’s long-term memory, and when?

The strongest answer: Only durable, useful facts and preferences the user stated, saved as short dated statements with their source, extracted after the turn, not in the reply path. Never every message, and no sensitive details by accident

Why it works: Right — memory is a write policy before it is a store. Durable, stated, dated, sourced; and the write happens off the latency path.

• Memory needs a write policy.
• Keep durable, stated facts and preferences.
• Record source and date; write asynchronously.
• Filter sensitive data before it lands.

You read a 40 GB log file to count matching lines. What shape does the code take?

The strongest answer: Iterate the file object line by line, or write a generator that yields matches — one line is in memory at a time, so the file size stops mattering

Why it works: Exactly. Streaming turns memory cost into a constant.

• Iterating a file yields lines lazily.
• Generators keep memory constant.
• readlines and read materialise everything.
• yield lets you compose stages without buffering.

URL shortener: how do you generate the short code?

The strongest answer: Either hash the long URL and take the first characters, handling collisions by retry, or hand out pre-generated unique codes from a counter or key service; say which, and why — a counter gives guaranteed uniqueness but predictable codes, a hash is stateless but needs collision checks

Why it works: Right. Name the scheme and its trade-off.

• Hash-and-truncate or pre-generated keys.
• Handle collisions explicitly.
• Predictability vs statelessness.
• Length from the target count.

Your assistant confidently describes an old version of a library that had a major release last month. What is going on, and what do you change?

The strongest answer: Its knowledge stops at a training cutoff and it cannot tell what has changed since. I put today’s date and current sources in the context, and have it flag what those sources do not cover

Why it works: Exactly — the fix is in the context: current date, current sources, and permission to say “I cannot confirm that”.

• Knowledge ends at the training cutoff.
• Give it the date and fresh sources.
• Let it say what it cannot confirm.
• A disclaimer is not a fix.

What is chunk overlap for, and what does too much of it cost?

The strongest answer: It keeps an idea that straddles a boundary whole in at least one chunk. Too much inflates the index and fills the top results with near-duplicates; splitting on headings and paragraphs first reduces the need

Why it works: Right — overlap is insurance against bad boundaries, and it has a premium: storage, embedding cost and duplicate hits that waste the context budget.

• Overlap protects ideas at boundaries.
• Too much means duplicates and cost.
• Split on structure before size.
• Test settings on your own documents.

Your assistant answers from company documents. Why show citations to the user, and what makes a citation worth showing?

The strongest answer: So users can verify a claim and decide how far to trust it. It is worth showing only if it opens the exact retrieved passage behind the sentence — from retrieval, not recalled by the model

Why it works: Exactly — a citation is a verification tool. One click to the supporting passage lets users check the answer, and catch the wrong ones for you.

• Citations exist so users can check.
• Link each claim to its passage.
• Build them from retrieval, not memory.
• A wrong citation costs more than none.

Your feature extracts supplier, date and total from invoices into JSON. A teammate proposes grading every eval case with an LLM judge.

What do you suggest instead?

The strongest answer: Code checks wherever the answer is checkable — valid schema, exact field match, numeric equality on the total — because they are cheap, fast and repeatable; keep model grading for open-ended qualities

Why it works: Exactly — use the most deterministic grader that can decide the case. A judge adds cost and noise where an equality check is already certain.

• Use the most deterministic grader that works.
• Schema and exact match in code.
• Judges only for open-ended quality.
• Similarity is not correctness.

What is a decorator, really?

The strongest answer: A function that takes a function and returns a replacement — @log above def f is just f = log(f), which is why the wrapper should carry functools.wraps

Why it works: Right, and wraps is the part people leave out until a name or docstring goes missing.

• @d over f means f = d(f).
• The decorator runs at definition time.
• functools.wraps keeps __name__ and __doc__.
• A decorator with arguments is one more layer.

Design a rate limiter: which algorithm and where does its state live?

The strongest answer: Token bucket or sliding window per client key, with the counters in a fast shared store like Redis so every gateway instance sees the same count; say what happens when the store is down — fail open or closed — because that is the real design decision

Why it works: Right. Shared, fast state, and a stated failure mode.

• Token bucket or sliding window.
• Shared fast store for counts.
• Keyed per client.
• Decide fail-open vs fail-closed.

What is a jailbreak, how is it different from prompt injection, and why does the difference matter to you as the engineer?

The strongest answer: A jailbreak is a user talking the model past its safety rules; injection is third-party text hijacking my app. The first calls for policy filters, the second for limits on what the model can reach

Why it works: Right — who attacks and who is hurt differ. A jailbroken chatbot says things it should not; an injected agent does things its user never asked for.

• Different attacker, different victim.
• Jailbreak: the user beats the model’s rules.
• Injection: outside text hijacks your app.
• Filters for one, least privilege for the other.

You need setup and guaranteed teardown around a block. What do you write?

The strongest answer: A context manager — @contextlib.contextmanager with a yield between setup and cleanup in a finally, or a class with __enter__ and __exit__

Why it works: Exactly, and the finally around the yield is what makes it hold on the error path.

• with guarantees cleanup on every exit.
• @contextmanager: setup, yield, cleanup in finally.
• Or __enter__ / __exit__ on a class.
• __exit__ can suppress an exception — usually it should not.

Design a news feed: fan-out on write or on read?

The strongest answer: Fan-out on write pushes each post into followers' precomputed feeds — fast reads, expensive for accounts with millions of followers; fan-out on read builds the feed at request time — cheap writes, slow reads; the usual answer is a hybrid: push for normal users, pull for the few huge ones

Why it works: Right. Hybrid, with the celebrity case named.

• Write fan-out: fast reads, costly writes.
• Read fan-out: the reverse.
• Hybrid for celebrities.
• Say the numbers that flip it.

A CPU-bound function is too slow. You add threads and nothing improves. Why?

The strongest answer: The GIL lets only one thread execute Python bytecode at a time, so threads help blocking I/O, not computation — use processes, a native library that releases the GIL, or move the work out of Python

Why it works: Exactly. Threads for waiting, processes for computing.

• The GIL serialises bytecode execution.
• Threads still win on I/O waits.
• multiprocessing for CPU-bound work.
• NumPy and friends release the GIL in C.

Your queue delivers at least once. What does that force on every consumer?

The strongest answer: Idempotency: each message carries a key and the consumer records what it has processed, so a redelivery after a crash or a slow ack is a no-op; you design the side effects to tolerate repeats rather than hoping for exactly-once

Why it works: Right. Repeats are certain; make them harmless.

• At-least-once means duplicates.
• Consumer keeps processed keys.
• Side effects tolerate repeats.
• Do not rely on exactly-once.

Transcripts of your support calls read well, but customer names, product codes and order numbers are often wrong — and those are what the downstream summary depends on.

How do you approach it?

The strongest answer: Those are speech-to-text’s known weak spots. Measure error on our own audio for those entities, not overall; add vocabulary hints where supported, correct against the catalogue and order system, improve the audio, and flag low-confidence segments

Why it works: Exactly — a good overall word error rate hides failures on the rare words that matter. Measure per entity, then constrain with the lists you already own.

• Average accuracy hides entity errors.
• Measure on your own audio, per entity.
• Constrain with known lists and hints.
• Flag low-confidence segments for review.

Your sales chatbot told a customer they were entitled to a discount that does not exist, and the customer is holding the company to it.

How do you make sure the assistant cannot commit the company to things that are not true?

The strongest answer: Take commitments out of free text: prices and policy come from the system of record via code, the model only explains them, and an output check blocks any amount or promise that does not match the source

Why it works: Exactly — constrain the output so the risky part is not generated at all, then verify what is left. The model explains; the system of record decides.

• Do not generate what must be exact.
• Numbers and policy come from code.
• Validate output against the source.
• Block or fall back on a mismatch.

One synchronous database call inside an async handler makes the whole service slow. Why?

The strongest answer: A blocking call stops the event loop, so no other coroutine can run until it returns — use an async driver, or push it to a thread with asyncio.to_thread

Why it works: Right. One blocking call in an async service is an outage waiting for load.

• Blocking code freezes the whole event loop.
• Use async drivers end to end.
• asyncio.to_thread for unavoidable blocking calls.
• One offender starves every other coroutine.

A hot cache key expires and 5,000 requests miss at once. Design against it.

The strongest answer: Coalesce: the first miss takes a lock or a single-flight slot and rebuilds while the others wait or serve the stale value; add early refresh before expiry with jittered TTLs so keys do not expire together; the database sees one read, not thousands

Why it works: Right. Single flight, stale-while-revalidate, jitter.

• One rebuild, others wait or serve stale.
• Refresh before expiry.
• Jitter TTLs.
• DB sees one read.

Your search matches queries to documents by averaging word2vec vectors of the words. Users searching “river bank” get pages about savings accounts.

What is the limitation, and what do you change?

The strongest answer: Static embeddings give one vector per word regardless of context, so both senses of “bank” share it and averaging blurs everything; a contextual model (BERT-style or a sentence-embedding model) encodes the phrase as a whole, sense included

Why it works: Exactly — one-vector-per-word cannot represent polysemy; contextual encoders produce different vectors for the same word in different sentences.

• Static: one vector per word.
• Polysemy collapses into one point.
• Contextual encoders embed the phrase.
• Sentence-embedding models for retrieval.

You switch the assistant to streaming. What changes for your backend beyond “the text appears sooner”?

The strongest answer: Only time to first token improves. Failures can now land mid-answer, so I handle partial output, cancel upstream when the client disconnects, and accept that I cannot validate or moderate the full reply before it shows

Why it works: Right — streaming moves the first byte, not the last. One success-or-failure becomes a sequence that can break halfway, and an abandoned stream can keep generating, and billing, unless you cancel it.

• Streaming moves first token, not last.
• Plan for failure mid-answer.
• Cancel upstream when the client leaves.
• Full-reply checks now happen after display.

Your assistant can call six tools. How do you evaluate whether it uses them correctly?

The strongest answer: Cases with an expected call: right tool — or none when none is needed — and valid, correct arguments, all checked in code, plus cases where it should ask for a missing detail rather than guess

Why it works: Exactly — selection, arguments and restraint are three separate failure modes, and all three are checkable without a judge.

• Grade the call, not only the answer.
• Right tool, or no tool.
• Arguments valid and correct.
• Include cases where it should ask.

Design a customer-support agent for a subscription product: it answers questions and handles simple account actions. Where do you start, and what are the parts?

The strongest answer: Scope and risk first: which intents it owns, what it must never do, when it hands off. Then grounded help-center answers, a few narrow tools scoped to the signed-in customer, confirmation before changes, and evals from real tickets

Why it works: Exactly — the design starts from what failure costs. Tools that act only as the signed-in customer mean a confused or manipulated agent cannot touch anyone else’s account.

• Scope and failure cost come first.
• Ground answers in the help center.
• Narrow tools, scoped to the customer.
• Confirm changes; hand off with context.

Three independent API calls run one after another in an async function. How do you overlap them?

The strongest answer: await asyncio.gather(a(), b(), c()) so all three are in flight at once — and decide what return_exceptions should do, because by default the first failure propagates

Why it works: Exactly, and the exception policy is the part usually left unconsidered.

• gather runs awaitables concurrently.
• Sequential awaits are sequential.
• return_exceptions decides fail-fast or collect.
• TaskGroup gives structured cancellation.

Placing an order touches inventory, payment and shipping services. How do you keep them consistent without one transaction?

The strongest answer: A saga: a sequence of local transactions with a compensating action for each step — reserve stock, charge, book shipping; if booking fails, refund and release; orchestrated by a coordinator or choreographed by events, with every step idempotent and the state persisted

Why it works: Right. Compensation, persisted state, idempotent steps.

• Local transactions with compensations.
• Orchestrated or choreographed.
• State persisted.
• Every step idempotent.

You are designing a Python service that fans out to many slow downstreams. What shapes the design?

The strongest answer: Async end to end so waiting costs a task rather than a thread, with bounded concurrency per downstream, timeouts on every call and a budget for the whole request

Why it works: Right, and the bound per downstream is what stops you from becoming their outage.

• Async suits many concurrent waits.
• Bound concurrency per dependency.
• Timeout every call, budget the request.
• Measure with real downstream latency.

Design multi-tenancy for a product where some tenants are regulated.

The strongest answer: Tiered isolation: shared schema with a tenant key and row-level enforcement for most, and a separate database or cluster for tenants whose contract or regulation demands it — one code path, an isolation level chosen per tenant, with noisy-neighbour limits and per-tenant keys for encryption

Why it works: Right. One product, several isolation tiers.

• Isolation as a per-tenant tier.
• Shared with row-level enforcement.
• Dedicated where required.
• Same code path.

A customer’s legal team asked your contract assistant whether a 300-page agreement had an auto-renewal clause. It answered “no”. There was one on page 212. You own the post-mortem.

What went wrong, and what do you change?

The strongest answer: Retrieval showed the model a few chunks, so “no” meant “not in what I read”. Absence and list-everything questions need a pass over every section or a clause index built at ingestion, and answers must state their coverage

Why it works: Exactly — top-k retrieval cannot prove a negative. Route exhaustive questions to a full pass, say what was checked, and put absence questions in the evals.

• Top-k retrieval cannot prove absence.
• Route exhaustive questions to a full pass.
• State coverage in every such answer.
• Add absence questions to the evals.

Six months after launch, how do you show that an AI feature is worth keeping?

The strongest answer: Against the user outcome we defined at the start — task done, time saved, tickets avoided — compared with a holdout or before-and-after, next to quality evals and cost per successful task. Usage alone does not count

Why it works: Exactly — outcome, comparison, cost. A feature people try and abandon looks healthy on a usage chart, and only a comparison tells you what changed because of it.

• Value means outcomes against a baseline.
• Define the outcome before launch.
• Compare with a holdout.
• Track cost per successful task.

Your worker cannot keep up with the producer and memory climbs. What is the right answer?

The strongest answer: Backpressure — a bounded queue that makes the producer wait or shed load, so the system degrades predictably instead of consuming memory until it dies

Why it works: Exactly. A bounded queue turns an outage into a slowdown you can see.

• Bounded queues create backpressure.
• Unbounded queues move the failure to memory.
• Decide explicitly: wait, shed or drop.
• Make the decision observable.

How do you limit the blast radius of a failure in a system serving millions?

The strongest answer: Cells: independent copies of the whole stack, each serving a slice of customers, with a thin routing layer; a bad deploy or a poison request takes out one cell, deploys roll cell by cell, and capacity grows by adding cells — at the cost of cross-cell features being hard

Why it works: Right. Isolation by cell; the cost is anything cross-cell.

• Independent full-stack cells.
• Customers pinned to a cell.
• Deploy cell by cell.
• Cross-cell features are the cost.

Every team has added tools to the shared company agent. It now has around sixty, several of them overlapping, and it increasingly picks the wrong one.

What do you do about it?

The strongest answer: Shrink what the model sees: merge overlapping tools, make descriptions say when to use each, load only the subset relevant to the task, and gate new tools behind a tool-selection test set with a named owner

Why it works: Exactly — it is an interface-design and governance problem. Fewer, sharper tools per request, and a test that tells you when a new tool breaks the old ones.

• Tool sprawl is an interface problem.
• Merge overlaps; sharpen descriptions.
• Show only the relevant subset.
• Test selection before adding tools.

Your offline eval score rose for three releases in a row. Over the same period, escalations from the assistant to human support went up.

What is going on, and what do you do?

The strongest answer: The offline set or its grader has drifted from production. I sample escalated sessions, label them, check the grader against those labels, fold them into the set, and keep an online signal beside the offline score

Why it works: Exactly — when the two disagree, the offline eval is the instrument under suspicion. Rebuild it from the sessions where users actually failed.

• When offline and online disagree, audit offline.
• Read and label the failing sessions.
• Validate the grader against humans.
• Refresh the set; keep both signals.

A Python service does heavy numeric work per request and latency is unacceptable. What options do you weigh?

The strongest answer: Push the loop into a library that releases the GIL, move the work to a process pool or a separate service, precompute or cache, or accept that this part does not belong in Python

Why it works: Right — and knowing when to leave Python for one component is part of the answer.

• Vectorised or native code releases the GIL.
• Process pools give real parallelism.
• Precompute or cache where the input repeats.
• Some components should leave Python.

Split a monolith into services without downtime — including the data.

The strongest answer: Strangler by capability: put a facade in front, carve one bounded context at a time, move its data by dual-writing into the new store while the monolith remains the source of truth, verify parity, flip reads, then writes, then delete; the database split is the real work, and the order of contexts is chosen by value and coupling

Why it works: Right. One context at a time, data migrated with parity checks.

• Facade, then one context at a time.
• Dual-write, verify, flip.
• Order by value and coupling.
• Delete the old path.

Every proposal that reaches you now says “use an LLM”: churn prediction from a customer table, demand forecasting, fraud scoring, and a support-ticket summariser.

How do you sort these, and what do you tell the teams?

The strongest answer: Match technique to data: tabular prediction and forecasting go to classical ML — cheaper, faster, measurable, explainable. Language work like the summariser suits an LLM. Each proposal brings a simple baseline first

Why it works: Exactly — generative AI is one branch of machine learning, not its replacement. A gradient-boosted model on a customer table is still the honest baseline to beat.

• Pick the technique the data calls for.
• Tabular and forecasting: classical ML first.
• Language tasks: LLM candidates.
• Demand a baseline before any build.

An enterprise customer in the EU will sign only if their data “stays in the EU”. Your product calls a hosted model, keeps embeddings in a vector store and traces every request.

What has to be true before you can say yes?

The strongest answer: Every stop on the path is in region: the inference endpoint, anything the provider retains, our vector store, traces, backups and eval sets — backed by contract, per-tenant regional routing, and honesty about any gap

Why it works: Exactly — residency is a property of the whole data flow, including the copies nobody thinks of. Map it, contract it, route it, and tell the customer plainly where you cannot meet it yet.

• Residency covers the whole data path.
• Map inference, retention, stores, traces, backups.
• Get regions and sub-processors in writing.
• Say plainly what cannot comply.