All jobs / Data science / ML / AI
Model Evaluation interview questions
Model Evaluation questions a Data science / ML / AI can be asked, from first job to senior. Open one to see the strongest answer and why it works.
Why do we split data into train, validation, and test instead of one pile?
The strongest answer: Train fits the model, validation guides choices I still control, test is a last look I do not tune against — two piles collapse those jobs
Why it works: Exactly. Three roles: fit, choose, report. Mixing them is how scores lie to you.
- Train fits; val chooses; test reports.
- Do not tune on test.
- Two piles merge choice and report.
- More data is not a substitute for a held-out look.
A teammate keeps rerunning the test set after each hyperparameter tweak. What do you say?
The strongest answer: Stop — that set is now another validation fold. Freeze a true holdout and only open it when the recipe is locked
Why it works: Exactly. Repeated peeking turns test into val. A holdout you keep opening is not a holdout.
- Repeated test peeks burn the holdout.
- Lock the recipe, then open test once.
- Do not average your peeks.
- Process beats “I will not look.”
Fraud is 1% of rows. The model is “99% accurate.” What is the problem?
The strongest answer: Accuracy can be a majority-class trophy — I ask what happens to the rare class and I name a metric that matches the decision
Why it works: Exactly. Quote the decision, then the metric. A 99% score on a 1% event can be “always say no.”
- Accuracy ignores class rarity.
- Name FN/FP costs, then the metric.
- A majority classifier can look “accurate.”
- No invented lift percentages.
You have daily sales for two years. How do you split?
The strongest answer: By time — train on the past, validate a later window, test the latest you still refuse to touch — not a shuffle that leaks tomorrow into today
Why it works: Exactly. Time-ordered data needs a time-respecting cut. Shuffle is future leakage.
- Temporal data: split in time.
- Do not shuffle days.
- Keep all kinds of days in the holdout.
- Shared periods are leakage.
When do you care more about recall than precision for a screening model?
The strongest answer: When missing a positive is worse than extra review — I still name who pays for the extras, I do not chant “recall is better”
Why it works: Exactly. The metric follows the cost of a miss versus a false alarm — and who does the follow-up.
- Match the metric to miss vs false-alarm cost.
- Name who reviews the extras.
- Neither metric is “more professional.”
- Do not pick the larger score.
The interviewer draws a 2×2 and asks where a false negative sits. Strongest answer?
The strongest answer: Predicted negative, actually positive — a miss. I say what that miss costs in this product, not just the cell name
Why it works: Exactly. Name the cell and the cost. Layouts differ; the meaning does not.
- FN = miss (actual + / predicted −).
- Tie the cell to a cost.
- FN ≠ FP.
- “Nobody noticed” is not a metric.
A few huge outliers dominate the loss. What do you say in an interview?
The strongest answer: I name the loss I chose and why — squared error punishes large misses harder; absolute or a robust loss if those rows are noise, not gold
Why it works: Exactly. The loss is a judgement about which mistakes matter. Defaults are not judgement.
- Pick a loss for the mistakes you care about.
- MSE emphasises large errors.
- Do not delete residuals for a prettier score.
- Do not hide behind R² alone.
A peer says “we optimise F1 because that is what ML people do.” Your reply?
The strongest answer: F1 balances precision and recall at one threshold — I use it only if that tradeoff matches the decision; otherwise I pick the metric that does
Why it works: Exactly. F1 is a specific compromise, not a membership card.
- F1 is one threshold tradeoff.
- Choose the metric for the action.
- Do not report a buffet to dodge the choice.
- No fake lift %.
Each user has 40 sessions. You split rows at random. What goes wrong?
The strongest answer: The same person leaks into train and test — I split on the user (or the real unit), not the row, unless I can argue independence
Why it works: Exactly. The unit of split is the unit of generalisation. People are not independent rows.
- Split on the real unit (user, device, day).
- Random rows can leak a person.
- Stratify ≠ de-identify.
- Do not copy entities across folds.
When is ROC-AUC a reasonable headline, and when is it a dodge?
The strongest answer: Reasonable as a ranking summary across thresholds; a dodge when the product uses one threshold and one cost I have not named
Why it works: Exactly. AUC can compare rankers. The product still needs a threshold and a cost.
- AUC summarises ranking.
- Still pick a threshold for the product.
- Do not treat AUC as purity.
- Do not call AUC a success %.
A slide says “model accuracy 99%.” What do you add or refuse?
The strongest answer: I add the class balance, the split, and the decision the number is for — or I refuse the slide. A bare 99 is a costume
Why it works: Exactly. A number without a denominator and a decision is marketing.
- Quote base rate, split, and decision.
- Refuse a bare 99%.
- No fake industry benchmarks.
- Decimals ≠ rigour.
Users appear across months. How do you validate a churn model?
The strongest answer: Hold out later months and whole users — I do not let last month’s behaviour of the same person sit in train while this month is test
Why it works: Exactly. Time and identity. Both can leak. A later window of new (or held-out) people is the adult cut.
- Hold out later time and users.
- Do not shuffle calendar blocks.
- Do not test the past with the future.
- One month is a snapshot, not a study.
Outreach can handle 200 accounts a week. What do you optimise?
The strongest answer: Precision at that capacity — or a ranked list I can cut at 200 — not a global F1 that assumes a different workload
Why it works: Exactly. The constraint is the metric. Capacity is a threshold.
- Optimise for the real capacity.
- A ranked cut beats a generic F1.
- Do not leave the threshold to ops unstated.
- No made-up lift.
Devices in one household share an address. You split by device id. Risk?
The strongest answer: Household behaviour still crosses the split — I split on household (or another shared key) when that is how the world repeats
Why it works: Exactly. Unique ids are not independent units. Shared life leaks.
- Split on the repeating real-world unit.
- Unique device ≠ independent.
- Shared keys can leak across devices.
- Encoding the key does not clean it.
The model says 0.9. Ops treats that as “90% sure.” What do you correct?
The strongest answer: A raw score is not a probability until I check calibration on a holdout — I say so, and I do not invent a confidence percent
Why it works: Exactly. Calibration is empirical. Softmax folklore is not a certificate.
- Check calibration before calling it a probability.
- Softmax ≠ frequentist 90%.
- Do not print fake confidence badges.
- Calibrate, then use the score.
Default 0.5 threshold on a 4% positive class. What do you change first?
The strongest answer: The threshold (and maybe the loss) using a validation slice and the real review capacity — 0.5 is a convention, not a law
Why it works: Exactly. Thresholds are product knobs. Conventions are not optima.
- Treat 0.5 as a default, not a law.
- Set the cut with costs and capacity.
- Base rate ≠ automatic threshold.
- Do not maximise accuracy to pick the cut.
How do you choose between a miss and a false alarm without fake money numbers?
The strongest answer: I name who is harmed and what work each error creates — review time, user trust, safety — then I pick a threshold with that story, not a made-up dollar
Why it works: Exactly. Costs can be qualitative and still decide. Invented dollars are theatre.
- Name who pays for each error.
- Qualitative costs still pick a threshold.
- Do not hide behind a department’s favourite error.
- No invented dollar values.
You used validation to pick a model and a threshold. What is left?
The strongest answer: A locked recipe scored once on test — if I hate the number I do not go back to shopping
Why it works: Exactly. One locked look. Shopping on test is the junior leak with more steps.
- Lock, then one test look.
- Do not shop on test.
- Val does not replace test.
- Do not blend the two scores.
You will search a large space of models and features. How do you keep the holdout honest?
The strongest answer: Nest the search inside training folds and keep an outer holdout — or a pre-registered confirmation set — so the search does not touch the number I report
Why it works: Exactly. Search is data use. Nest or pre-register. Shopping the outer scores is the same old peek.
- Nest search; protect an outer look.
- Watching a set spends it.
- Do not search on all rows.
- Do not shop outer-fold winners.
Global AUC is fine; one country is a disaster. What do you present?
The strongest answer: The slices I promised to watch — volume and the metric that matches the decision — and I do not average them away
Why it works: Exactly. The product fails in a place. Averages hide places.
- Show the slices that can fail.
- Do not hide them in a global average.
- Do not omit a failing country.
- No invented fairness %.
A model beats the global baseline and loses on new customers. What do you ship?
The strongest answer: A policy that uses the model where it wins and the baseline where it does not — or I do not ship a global flip
Why it works: Exactly. Segmented policy is allowed. A global promise is not a reason to hurt a slice.
- Route by slice if the win is local.
- Do not global-flip over a hurting segment.
- A weak slice is not “delete the model.”
- Do not coin-flip two scores.
Three teams report three “the” model metrics. What do you standardise?
The strongest answer: The decision, the primary metric, the split rule, and who can change them — I do not standardise every notebook library
Why it works: Exactly. Shared language and change control. Local tooling can differ.
- Standardise decision, metric, split, owners.
- Let libraries differ.
- Do not become the approver of every trial.
- Do not write a shelf document.
Risk wants fewer false approvals; growth wants more volume. How do you refuse a single score?
The strongest answer: I put both costs on one decision record and make them pick a threshold together — a secret average is how both sides feel lied to
Why it works: Exactly. One decision, two costs, one explicit cut. Averages and dual promises are politics.
- One explicit shared threshold.
- Write both costs down.
- Do not hide in a “balanced” score.
- No twin invented wins.
The company optimised click-through for a year and trust fell. Your postmortem line?
The strongest answer: We let a proxy become the goal — the next primary metric must include the harm we ignored, and we say that in the write-up
Why it works: Exactly. Name the Goodhart. Do not blame users or invent a recovery lift.
- Write the proxy failure down.
- Change the primary to include the harm.
- Do not blame users.
- No invented recovery lift.
The board wants one number for “is the AI working?” What do you give?
The strongest answer: The live decision metric we pre-agreed, plus one sentence on residual risk — I do not invent an “AI health %”
Why it works: Exactly. One agreed live metric and the leftover risk. Crowns and losses are not the business.
- One pre-agreed live metric.
- One sentence of residual risk.
- No invented health %.
- No vendor crown as the answer.