All jobs / Data science / ML / AI
MLOps interview questions
MLOps questions a Data science / ML / AI can be asked, from first job to senior. Open one to see the strongest answer and why it works.
You are running dozens of training experiments. What do you record for each run, and why?
The strongest answer: Code commit, data version, hyperparameters, environment, seed, and the metrics and artefacts — in a tracker such as MLflow or Weights & Biases — so any run can be reproduced and compared
Why it works: Right — an experiment is code plus data plus configuration plus environment; miss any one and the number cannot be reproduced.
- Record code, data, params, env, seed.
- Log metrics and artefacts per run.
- Use a tracker, not filenames.
- Goal: reproduce and compare.
The only artifact is a notebook that “ran once on my laptop.” What is missing?
The strongest answer: A recipe someone else can rerun — data version, seed, environment, and the exact split — not a screenshot of a cell
Why it works: Exactly. Reproducibility is the minimum adult step. Screenshots are souvenirs.
- Pin data, seed, env, split.
- Someone else must be able to rerun.
- A screenshot is not an artifact.
- Export ≠ model card.
A junior changes the system prompt in production because a demo failed. What do you want them to learn?
The strongest answer: Prompts are code — version them, eval them, and roll back — a live tweak to “sound nicer” is an unreviewed release
Why it works: Exactly. If it changes behaviour, it is a release. Treat it like one.
- Version and eval prompts.
- A prompt tweak is a release.
- There is no “forget” rollback.
- Do not crowd-tune production prompts in public.
What do you monitor besides “accuracy last month” in production?
The strongest answer: Input and prediction shift, volume, latency, and a fresh-label slice when I can get one — a stale accuracy is a souvenir
Why it works: Exactly. The world moves. Labels lag. Watch the inputs and the outputs, then buy labels on purpose.
- Watch inputs, outputs, volume, latency.
- Sample fresh labels on purpose.
- Stale accuracy is a souvenir.
- No invented trust percentiles.
Offline AUC went up. Product conversion did not. What do you say?
The strongest answer: The offline proxy did not match the decision or the traffic — I debug the proxy, the slice, and the launch, I do not invent a “delayed lift”
Why it works: Exactly. Offline is a rehearsal. When the show differs, believe the show and fix the rehearsal.
- Offline ≠ online.
- Debug proxy, slice, and launch.
- No invented delayed lift.
- Do not retarget the metric to save the model.
Two of you cannot reproduce the same AUC. First place you look?
The strongest answer: Whether you trained on the same data version, split, and filters — then code and seeds — not “who is the better tuner”
Why it works: Exactly. Data and split first. Ego last.
- Pin dataset version and split.
- Then code and seeds.
- Do not average two recipes.
- Do not keep only the higher score.
What training-serving skew will you not tolerate as a staff default?
The strongest answer: Different joins, clocks, or default-missing logic between train and serve — I treat that as a defect, not as “batch versus online flavour”
Why it works: Exactly. Same meaning of a feature at score time. Bit-identical is a slogan; silent different joins are a bug.
- Same joins, clocks, missing-logic.
- Skew is a defect, not a flavour.
- Do not watch only AUC.
- Do not empty the model to agree.
What belongs next to a shipped score in the README?
The strongest answer: Who it is for, when the features are known, what the score is not, and how it can fail — not only the AUC
Why it works: Exactly. Assumptions and failure modes. A screenshot is not a contract.
- Document scope, timing, and non-goals.
- Name failure modes.
- AUC alone is not a README.
- No fake industry comparisons.
How do you tell covariate shift from concept drift in a sentence?
The strongest answer: Covariate: the inputs we see changed. Concept: the relationship from input to outcome changed. I do not treat every monitor blip as the same fix
Why it works: Exactly. Different medicine. Retrain, relabel, or change the product — depending on which one it is.
- Covariate = input mix changed.
- Concept = the mapping changed.
- Different fix for each.
- Do not treat every blip as “need a bigger net.”
The feature store serves “current” values in training jobs. What do you require?
The strongest answer: Point-in-time correct training datasets — current-in-prod is not current-in-history. I refuse a store that cannot travel
Why it works: Exactly. Training on today’s features for yesterday’s labels is a time machine. The store must serve as-of.
- Demand point-in-time training.
- “Current” during backfill is leakage.
- Yearly snapshots are not as-of joins.
- Do not mix clocks between train and eval.
A new ranker might scramble a money path. How do you launch?
The strongest answer: Shadow on live traffic, compare decisions, then a small canary with a kill switch — I do not flip 100% on a better offline score
Why it works: Exactly. Watch, then a little, then more. Offline is not a traffic cop.
- Shadow, then canary, then more.
- Keep a kill switch.
- Offline wins ≠ 100% flip.
- Do not launch into the weekend by habit.
Input distributions look stable; precision collapsed. What do you check?
The strongest answer: Whether the meaning of the label moved — policy, fraudsters, or annotators — not only the feature histograms
Why it works: Exactly. Concept and label processes move while features sit still. Histograms are not the whole patient.
- Label meaning can move alone.
- Check policy and annotators.
- Feature histograms are not enough.
- Do not smooth away a collapse.
Drift alerts fire daily and nobody looks. What do you change?
The strongest answer: The alert: a threshold tied to an action and an owner — I do not add a second dashboard
Why it works: Exactly. An alert without an action is decoration. Owners and thresholds, or delete the noise.
- Alerts need an owner and an action.
- Do not add wallpaper dashboards.
- Do not page leadership for noise.
- Do not mute without redesigning.
A 2-point offline gain costs 900 ms on the search path. How do you decide?
The strongest answer: With the product: whether users feel the delay and whether the gain shows up on the live decision — I do not treat offline points as sacred
Why it works: Exactly. Latency is part of the quality users get. Sacred offline points are how you ship a slower miss.
- Latency is user-facing quality.
- Decide with the live outcome.
- Do not split “quality” from “ops.”
- No fake combined lift.
Training used a join that production cannot run in budget. What failed?
The strongest answer: The feature contract — I define features as something serving can compute, with owners and SLOs, before I fall in love with the offline join
Why it works: Exactly. If you cannot serve it, it is not a feature. It is a notebook column.
- Features must be servable.
- Write owners and SLOs.
- Do not hide an unservable join in the client.
- Do not advertise an offline-only score.
You inherit a “winning” stack: store, training, batch, serve. First senior pass?
The strongest answer: A leakage and time-travel review across joins, labels, and training-serving skew — I do not start by retuning hyperparameters
Why it works: Exactly. Wins that cannot survive a clock are not wins. Frameworks do not fix a join.
- Review time-travel and skew first.
- Do not start with a retune.
- Do not scale a leak.
- Do not polish a leaked score.
The generator starts producing unsafe instructions. What must already exist?
The strongest answer: A named person who can disable the path in minutes, a fallback, and a comms line — a Slack thread is not a kill switch
Why it works: Exactly. Minutes, owner, fallback. Tomorrow’s fine-tune is not an incident tool.
- Owner, minutes, fallback, comms.
- Prompts ≠ kill switch.
- Retrain is not incident response.
- Do not wait for the vendor.
What is your rule for what may be promoted to production in the model registry, and what does the registry entry have to point to?
The strongest answer: Promotion is a gated event: the entry links the exact training run, data version, evaluation on the frozen holdout and slices, the approval and the rollback target; serving pulls by version, never by “latest”
Why it works: Exactly — a registry is lineage plus a promotion contract; “latest” in serving is how an untested model reaches users.
- Promotion is a gated event.
- Link run, data, eval, approval.
- Rollback target is part of the entry.
- Serve by version, never latest.
You want to train on production outcomes. What do you lock first?
The strongest answer: How those outcomes were produced — policy, UI, and the previous model — so you are not cloning a past decision as if it were nature
Why it works: Exactly. Production labels are decisions. Document the policy that made them, or you will fossilise it.
- Lock the policy that created the label.
- Prod outcomes are not nature.
- Do not keep only agreeing rows.
- Do not relabel to flatter the model.
What is the minimum reproducibility bar you will sign?
The strongest answer: Same data version, same code digest, same config, and a second machine can rebuild the artifact — “it worked on my GPU” is not a bar
Why it works: Exactly. Artifact rebuild. HTML and calls are souvenirs.
- Data, code, config, rebuild.
- Souvenirs are not the bar.
- Pin the environment.
- Do not accept “worked on my GPU.”
A ranking model is sending people to a harmful result. First hour?
The strongest answer: Disable or constrain the path, preserve logs, tell the people who must know, then diagnose — I do not start with a research postmortem while it still serves
Why it works: Exactly. Stop the harm, keep the evidence, communicate. Retrain is later.
- Stop or constrain first.
- Preserve logs; notify.
- Do not panic-retrain.
- Do not keep harm up “for data.”
An auditor asks which rows trained last Thursday’s build. What must exist?
The strongest answer: Lineage from the artifact back to a data version and a filter — a folder of CSVs named final_final is not lineage
Why it works: Exactly. Artifact to snapshot. Memory and screenshots fail the second question.
- Artifact → data version → filter.
- Memory is not lineage.
- Screenshots are not digests.
- No invented sample % as a defence.
What is the staff-level launch policy you will defend?
The strongest answer: Shadow until disagreement is understood, canary with a kill switch, expand on live outcomes — not on a prettier offline deck
Why it works: Exactly. Understand, then a little, then more. Decks and seniority are not traffic.
- Shadow → canary → expand on live outcomes.
- Kill switch required.
- Offline gain is not a flip.
- Do not hide canaries in quiet regions.
A paper-quality model has no serving path and a leaky split. What do you tell leadership?
The strongest answer: It is research until it has a leak-safe recipe and a serve path — I will not relabel it “ready” to protect a demo date
Why it works: Exactly. Ready has a clock and a path. Demos are allowed; lies about ready are not.
- Research ≠ production.
- Need leak-safe recipe + serve path.
- Do not ship to protect a date.
- No invented launch percentile.
A legacy ranker still serves 8% of traffic “because it is stable.” When do you retire it?
The strongest answer: When nobody can explain its features, replay its labels, or own its incidents — stability without lineage is a pet
Why it works: Exactly. If you cannot replay or own it, you cannot defend it. Pets accumulate incidents.
- Retire what you cannot replay or own.
- Stability without lineage is a pet.
- Do not keep it because it is old.
- Do not Friday-kill to find owners.
How should champion/challenger run so it is not a beauty contest?
The strongest answer: Same traffic rules, same primary metric, a pre-written promote/kill bar, and no extra peeks for the favourite team
Why it works: Exactly. A fair exam and a written bar. Favourites and eternal averages are politics.
- Same rules, same metric, written bar.
- No gifted slices.
- No notebook beauty contest.
- Do not average forever to avoid a call.