All jobs / AI engineer

Production AI Engineering interview questions

Production AI Engineering questions a AI engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.

1. Why does a call to a model need a timeout, and what should the user see when it fires?

   The strongest answer: Model calls can hang or run very long, and a request that waits forever holds a connection and a user; set a limit, then show a clear message or a simpler fallback, and log it

   Why it works: Exactly — the timeout protects your servers and the fallback protects the user. Both are decided before the first slow day, not during it.

   • Every external call gets a time limit.
   • Set it from what users will tolerate.
   • Show an honest message or a fallback.
   • Log every timeout.

2. Many providers offer a moving alias that always points at their newest model, and fixed dated versions. Which do you use in production, and why?

   The strongest answer: A fixed version: behavior then changes only when we choose, after re-running our evals. A moving alias can change the answers overnight with no deploy on our side to explain it

   Why it works: Exactly — pinning turns a model change into a release you control: evaluate, switch, and keep the old version to roll back to.

   • Pin the model version in production.
   • A moving alias changes behavior silently.
   • Re-run the evals before every switch.
   • Watch the retirement date of your pin.

3. During a provider slowdown your model bill for the hour tripled. The SDK retries three times, your service wrapper retries three times, and the queue re-delivers a failed job three times.

   What is going on, and what do you change?

   The strongest answer: The layers multiply — one job can become 27 calls, and a call we gave up on may still be billed. Retry in one layer only, with backoff and jitter, on retryable errors, within a budget

   Why it works: Exactly — retries belong to one owner, and that owner has a budget. Work the provider already started can cost money even after you stopped listening.

   • Retries in layers multiply.
   • One layer owns the retries.
   • Backoff, jitter, retryable errors only.
   • Cap them with a retry budget.

4. One customer starts a bulk import that fires thousands of model calls. Within minutes every other customer’s chat is failing with HTTP 429 from the provider.

   What is missing from your system, and what do you add?

   The strongest answer: Our own limits in front of the provider’s: per-tenant quotas counted in tokens as well as requests, and bulk work on a lower-priority queue, so one customer cannot spend the shared quota that interactive traffic needs

   Why it works: Exactly — the provider’s limit is shared by your whole product, so fairness is your job. Tokens matter because one large request can use as many as a long run of small ones.

   • The provider quota is shared; ration it.
   • Limit per tenant, in tokens too.
   • Separate bulk from interactive traffic.
   • Interactive work gets priority.

5. The model provider is not down, but responses that took two seconds now take forty. Your API’s worker pool fills with waiting calls, and unrelated endpoints start failing too.

   What pattern stops this, and what does the AI feature do while it is active?

   The strongest answer: A circuit breaker around the model dependency, plus a tight timeout and its own bounded pool: after repeated failures it fails fast, the feature serves a degraded mode, and probe calls decide when to resume

   Why it works: Exactly — slow is worse than down, because nothing fails until everything does. The breaker turns slow into fast failure, and the degraded mode was designed in advance.

   • A slow dependency takes everything down.
   • Isolate model calls in their own pool.
   • Trip the breaker; fail fast.
   • Design the degraded mode in advance.

6. The monthly model bill doubled, and nobody can say which customers or which features caused it. What do you build so that this cannot happen again?

   The strongest answer: Tag every model call with tenant, user and feature at the gateway, meter tokens and cost per tag, set budgets with alerts and hard caps, and define what each feature does when its cap is reached

   Why it works: Exactly — attribution first, then budgets, then a planned behavior at the cap. A runaway loop or one heavy tenant shows up the same day, not on the invoice.

   • You cannot budget what you cannot attribute.
   • Tag calls: tenant, user, feature.
   • Budgets with alerts and hard caps.
   • Decide the behavior at the cap.

7. Your provider announces that the model version you pinned will be retired in a few months. Forty prompts across six features depend on it.

   How do you run the migration?

   The strongest answer: As a release: inventory the prompts, run each feature’s evals on the candidate, fix the prompts that regress, then roll out feature by feature behind a flag, with the old pin as rollback while it still exists

   Why it works: Exactly — and the features with no eval suite are the real finding: start there, because without one you are migrating blind.

   • A model change is a release.
   • Inventory the prompts; run the evals.
   • Roll out per feature behind a flag.
   • Migrate early, while rollback still exists.

8. Offline evals say a cheaper model is about as good as the current one for your support assistant. Product wants to switch everyone tomorrow to cut cost.

   How do you decide with evidence from production?

   The strongest answer: An A/B test: assign by user so each person stays on one model, agree success and guardrail metrics up front — resolutions, escalations, latency, cost — log the variant on every trace, and keep a kill switch

   Why it works: Exactly — offline evals say it is safe to try; the experiment says what happens with real users. Metrics chosen in advance keep you honest about the result.

   • Offline says try; online says switch.
   • Assign per user, not per request.
   • Fix success and guardrail metrics first.
   • Log the variant; keep a kill switch.