Open-Source & Local Models interview questions
Open-Source & Local Models questions a AI engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
What is the difference between an “open-weight” model and an open-source one, and why does it matter to you as an engineer?
The strongest answer: Open-weight means the trained weights can be downloaded, under a license that may still limit commercial use; open source would also open the training code and data. So I read the license first
Why it works: Exactly — weights you can run are not the same as a model you can reproduce or use without conditions. The license decides what your product may do.
- Downloadable does not mean unrestricted.
- Open-weight: the weights are published.
- Open source: code and data too.
- Read the license before you build.
What does a bigger model buy you, what does it cost, and how do you choose a size?
The strongest answer: Usually more ability on hard reasoning and broader knowledge, paid for in memory, latency and cost per token. I pick the smallest model that passes my eval for the task
Why it works: Right — capability against memory, speed and money, settled by your own eval. Plenty of narrow tasks are handled well by a small model.
- Smallest model that passes the eval.
- Bigger: more capable, slower, costlier.
- Size is not context length.
- Let your eval choose.
What is quantization, and what do you give up for it?
The strongest answer: Storing the weights at lower numeric precision, so the model needs less memory and often runs faster. Some quality is lost, more as precision drops, so I re-run my eval on the quantized model
Why it works: Exactly — fewer bits per weight. Moderate quantization often costs little and aggressive quantization can hurt, and the damage depends on the task, so you measure.
- Fewer bits per weight, some quality lost.
- Less memory, often more speed.
- Loss grows as precision drops.
- Re-run the eval after quantizing.
A prototype talks to a model on a developer’s laptop through a local runner such as Ollama. The team wants to put the same setup in front of a few hundred users.
The team wants to serve real users from the laptop prototype’s local runner. What do you tell them?
The strongest answer: A local runner is built mainly for one person’s convenience; many concurrent users need a serving engine built for throughput — batching, queuing, metrics — such as vLLM. I’d load-test it at the expected concurrency
Why it works: Right — same model, different job. Development tools optimize for ease, serving engines for concurrency, and only a load test tells you what the setup really holds.
- A dev runner is not a serving engine.
- Local runner: one user, easy setup.
- Serving engine: batching, queues, metrics.
- Load-test at real concurrency.
A self-hosted model loads fine on the GPU and answers short prompts. With long documents, or several users at once, it runs out of memory.
A self-hosted model runs out of GPU memory only on long prompts or with several users. Why, and how do you plan capacity properly?
The strongest answer: Weights are only part of it: the KV cache grows with context length and with concurrent sequences. I budget weights plus cache for the longest context and the concurrency I must serve, and cap both
Why it works: Exactly — the model “fitting” is only the start. Serving memory is the weights plus per-token cache for everything in flight, and limits on context and concurrency keep you inside it.
- Serving memory: weights plus KV cache.
- Cache grows with context length.
- And with concurrent requests.
- Budget for both; cap both.
You serve a model through an inference server such as vLLM. What does continuous batching give you, and what is the trade-off?
The strongest answer: It runs many requests together and admits new ones as others finish, so the GPU stays busy and throughput rises. Each request slows under load, so I set concurrency limits against a latency target
Why it works: Right — throughput and cost per token improve, and per-request latency is what you pay. Measure time to first token and tokens per second separately, under realistic load.
- Batching trades latency for throughput.
- Requests share the GPU continuously.
- Cap concurrency to a latency target.
- Measure first-token time and tokens per second.
Leadership asks whether to self-host an open-weight model or keep paying for a hosted API. How do you frame the decision?
The strongest answer: By requirements: data constraints, the quality bar on our eval, how much volume and how steady, latency, and whether we can run GPUs on call. Self-hosting swaps per-token fees for fixed capacity and ops work
Why it works: Exactly — it is a total-cost and capability question. Self-hosting tends to win on steady heavy volume or hard data rules, and to lose on spiky load and small teams.
- Frame by requirements and total cost.
- Data rules, quality bar, volume, latency.
- Count GPUs, ops and upgrades.
- Steady heavy load favors self-hosting.
A customer in a sensitive sector writes into the contract talks: “the model must run locally, for privacy.”
A customer insists that the model must run locally “for privacy”. What do you do with that requirement?
The strongest answer: Find the requirement under it — where data may go, who may process it, what is kept. Local inference removes the outside processor but leaves logs, access and patching to us, and may cost quality
Why it works: Right — sometimes a no-retention or in-region hosted endpoint meets the real need, sometimes only on-premises does. Either way the trade in quality and operations is stated up front, not discovered later.
- “Local” is a means; find the requirement.
- Ask where data may go and stay.
- Local still needs logs, access, patching.
- State the quality trade-off honestly.