LLM APIs interview questions
LLM APIs questions a AI engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
Your chat feature sends only the user’s newest message to the model API, and follow-up questions get confused replies. What is going on, and what do you change?
The strongest answer: Each API request stands alone — the model sees only what that request contains, so my code must send the earlier turns (or a stored-conversation reference, where the provider offers one) every time
Why it works: Exactly — the model holds nothing between calls. Whatever it should know, this request has to carry, which is also why long chats cost more per turn.
- Every request stands alone.
- Send the turns the model needs.
- History is input — you pay for it.
- Read your payload before blaming the provider.
How do you choose which model to call for a new feature?
The strongest answer: Start from the task: collect a few real test cases, prove it on a capable model, then see whether a smaller, cheaper, faster one passes the same cases within our latency and data constraints
Why it works: Right — the task and your own cases decide, not the brand. Proving it on a strong model first tells you whether a failure is the model or the prompt.
- Your test cases choose the model.
- Prove it on a capable model.
- Then try smaller, faster, cheaper.
- Weigh quality, latency, cost, data rules.
A service asks the model for an order as JSON and deserialises the reply straight into the order object. Now and then the parse fails, and once it created an order for a product that does not exist.
What is missing between the model’s reply and your order object?
The strongest answer: Validation. I use schema-constrained output where the provider has it, then check the parsed object in code — types, ranges, ids that must exist — and treat truncated or refused replies as an error path
Why it works: Exactly — a schema makes the shape reliable, not the content. Model output is untrusted input like any other, and a reply cut off at the output limit is not valid JSON.
- Model output is untrusted input.
- Constrain the shape with a schema.
- Validate values and ids in code.
- Truncation and refusal are error paths.
At peak your service starts getting HTTP 429 from the model provider. What should the client do?
The strongest answer: Treat it as “slow down”: retry with exponential backoff and jitter, honour a retry-after hint if one is sent, cap the attempts, and retry only what is retryable — 429, timeouts, 5xx — never a 400
Why it works: Exactly — back off, add jitter so clients do not retry in step, and never retry an error that will fail the same way. Then ask why you are hitting the limit at all.
- 429 means slow down, not stop.
- Exponential backoff with jitter, capped.
- Honour retry-after when it is sent.
- Retry only errors that can succeed.
You switch the assistant to streaming. What changes for your backend beyond “the text appears sooner”?
The strongest answer: Only time to first token improves. Failures can now land mid-answer, so I handle partial output, cancel upstream when the client disconnects, and accept that I cannot validate or moderate the full reply before it shows
Why it works: Right — streaming moves the first byte, not the last. One success-or-failure becomes a sequence that can break halfway, and an abandoned stream can keep generating, and billing, unless you cancel it.
- Streaming moves first token, not last.
- Plan for failure mid-answer.
- Cancel upstream when the client leaves.
- Full-reply checks now happen after display.
Traffic to the assistant grew by a tenth this month and the model bill doubled. How do you find out why, and what do you expect to find?
The strongest answer: Log the input and output token counts each response reports, tagged by feature and prompt version, and see which grew. Often it is input — longer history, more retrieved context, a fatter system prompt, or retries
Why it works: Exactly — you pay for tokens, not requests, and the usage figures come back with every call. Find the feature whose tokens per request grew, then trim, cache or cap that one.
- You pay per token, not per request.
- Log usage by feature and prompt version.
- Look first at input growth.
- Fix the cause, then consider cheaper models.
A nightly job classifies the day’s support tickets through the same synchronous endpoint and API key as the live chat product. Chat users now see rate-limit errors every evening.
What is your call?
The strongest answer: Separate the workloads: bulk goes through a queue with its own quota and concurrency cap — or the provider’s batch interface, if it has one — on the smallest model that passes a labelled sample
Why it works: Right — interactive and bulk traffic should not share one fate. Bulk work tolerates delay, so it can queue, run cheaper and yield, and a labelled sample tells you how small the model can go.
- Interactive and bulk never share one fate.
- Queue bulk work with its own quota.
- Use batch interfaces where they exist.
- Smallest model that passes a labelled sample.
The assistant’s p95 response time is 9 seconds. Product wants it under 3 and asks you for a date.
What are your levers, in what order, and what do you promise?
The strongest answer: Measure where the time goes first. Then shorten the output, stream, trim or cache the prompt, send easy requests to a smaller model — and promise a date only after the eval set shows the quality cost
Why it works: Exactly — output is generated token by token, so the length of the answer often dominates, but you only know after measuring. Every lever trades quality, and the eval set is how you price it.
- Measure before you pull any lever.
- Output length often dominates generation time.
- Stream, trim, cache, route the easy requests.
- Price every lever on the eval set.