All jobs / AI engineer

AI Application Architecture interview questions

AI Application Architecture questions a AI engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.

1. What is an AI gateway, and why would a team put one between its services and the model providers?

   The strongest answer: One internal service every model call goes through, so keys, logging, budgets, rate limits and model routing live in one place instead of being copied into every feature

   Why it works: Exactly — one choke point means one place to see spend, rotate a key, enforce a limit or swap a model.

   • One door for every model call.
   • Keys and limits live in one place.
   • Log and meter every call there.
   • Swap or route models without touching features.

2. The model API is stateless, and your chat service runs on several servers. Where does the conversation live between messages?

   The strongest answer: In a shared store such as a database, keyed by conversation id: whichever server gets the message loads the history, sends the relevant turns to the model, and saves the new ones

   Why it works: Exactly — the model only knows what this request contains, so memory is your application’s job, and a shared store keeps every server interchangeable.

   • The model is stateless; you hold the memory.
   • Keep conversations in a shared store.
   • Load, send the relevant turns, save.
   • Keep the app servers stateless too.

3. A “summarize this report” endpoint calls the model inline, and large reports take longer than the load balancer’s 60-second limit. How do you redesign it?

   The strongest answer: Accept the upload, put a job on a queue and return a job id at once; a worker calls the model and stores the result, the client polls or is notified, and the job is safe to run twice

   Why it works: Exactly — the queue gives you durability, backpressure and retry control, and the user gets an honest “working on it” instead of a dead connection.

   • Slow model work leaves the request path.
   • Enqueue, then return a job id.
   • A worker calls the model, stores the result.
   • Make the job safe to run twice.

4. You want to cache model responses to cut cost. What goes into the cache key, and what should you not cache at all?

   The strongest answer: Hash everything that shapes the output — model, prompt version, parameters and the full input — and scope it per user or tenant where the input is personal; skip anything that must be fresh or deliberately varied

   Why it works: Exactly — leave out an input that changes the answer and you serve a wrong answer; leave out the tenant scope and you serve someone else’s.

   • The key is everything that shapes output.
   • Include model, prompt version, parameters, input.
   • Scope personal answers per user or tenant.
   • Do not cache fresh or creative output.

5. Your semantic cache matches new questions to stored answers by embedding similarity. A user asking “How do I cancel my annual plan?” was served the stored answer for “How do I cancel my monthly plan?”.

   What went wrong, and how do you make the cache safe?

   The strongest answer: Similar is not the same — close embeddings, different answers. Raise the threshold, add structured keys such as plan and tenant, limit it to stable FAQ-type questions, and measure false hits on a labeled set

   Why it works: Exactly — a semantic cache trades correctness for cost, so you bound where it may answer and you track the false-hit rate like any other quality metric.

   • Similar questions can need different answers.
   • Add hard keys for answer-changing facts.
   • Cache only stable, low-risk intents.
   • Measure false hits on labeled pairs.

6. You put every provider behind one internal “complete(prompt)” interface so that models are swappable. Where does that abstraction leak in practice?

   The strongest answer: Prompts are tuned per model, and tool calling, structured output, token limits, error types and safety behavior all differ — so a swap needs a per-model adapter and an eval run, not only a config change

   Why it works: Exactly — the interface hides the wire format, not the behavior. The honest version of the abstraction is “swappable after the evals pass”.

   • An interface hides format, not behavior.
   • Keep prompts and adapters per model.
   • Expose capabilities; do not fake them.
   • Swap only after the evals pass.

7. Most of your traffic is simple classification and a small share is hard reasoning, yet everything goes to the most expensive model. How do you design routing, and how do you know it is not quietly hurting quality?

   The strongest answer: Route by task type first, because the caller knows it; give easy tasks a cheaper model, escalate on failed validation, and keep per-route evals and sampled production comparisons so the saving is measured against quality

   Why it works: Exactly — the cheapest reliable signal is the one you already have, and a router is a quality risk until each route has its own eval.

   • Route on signals you already have.
   • Cheap model for easy, known tasks.
   • Escalate when validation fails.
   • Evaluate every route separately.

8. After a provider outage took your assistant down for two hours, leadership asks for “a fallback model on another provider”.

   What has to be true before that fallback is real rather than a line in a config file?

   The strongest answer: Prompts and tool schemas are adapted and evaluated on the fallback model, the data agreements cover sending it that traffic, quota exists for the load, and the switch is exercised regularly rather than first tried mid-outage

   Why it works: Exactly — a fallback you have never evaluated or exercised is a hope. Quality, legal cover, capacity and a rehearsed switch make it a plan.

   • An untested fallback is not a fallback.
   • Evaluate your prompts on the fallback model.
   • Check data terms and quota first.
   • Rehearse the switch before you need it.