Experimentation interview questions
Experimentation questions a Product manager can be asked, from first job to senior. Open one to see the strongest answer and why it works.
What must an experiment state before you run it?
The strongest answer: The question, the primary measure, and what you will do if it fails — not “let’s A/B and see”
Why it works: Exactly. Question, measure, fail path. Promised winners and buffet metrics are how every test “wins.”
- Question + primary measure + fail action.
- Not “see what happens.”
- No metric buffet.
- No invented lift target.
What is a demand test — a landing page or a button for a feature that does not exist yet — actually measuring, and what must you not do with it?
The strongest answer: Interest at that moment, from the people who happened to see it. A click is attention, not a promise to pay, and whoever clicks must get an honest message rather than a charge or a dead end
Why it works: Exactly — a click is evidence of attention from a specific audience. The honest page says the feature is not ready and offers to tell them when it is.
- A click is attention, not demand.
- Name who saw it.
- Tell the clickers the truth.
- Never take money for vapour.
What makes an experiment honest after it starts?
The strongest answer: A stop rule you wrote first — I do not peek daily and stop when the line is pretty
Why it works: Exactly. Pre-written stop. Peeks, mid-flight metric swaps, and early lifts are the same cheat.
- Write the stop rule first.
- Do not peek-to-pretty.
- Do not swap the primary mid-flight.
- Do not announce lift early.
What is an MVP allowed to be?
The strongest answer: The smallest thing that can answer the question — not a half-ugly version of the final vision
Why it works: Exactly. Answer the question. Half-of-everything, beta-final, and 3× promises miss the point.
- MVP answers a question.
- Not half of the vision.
- Not the vision with a badge.
- No 3× promise on a landing page.
You will try three copy variants and two metrics. How do you stay honest?
The strongest answer: Write the primary and the confirmation window before the hunt — extras are labelled exploration
Why it works: Exactly. Pre-specify. Hunting and launch-post buffet are how every test “works.”
- Primary + window first.
- Label extras as exploration.
- Do not pick the metric after peeking.
- Do not publish every hunted win.
When do you write kill criteria?
The strongest answer: Before you fall in love — what evidence would stop the bet — I do not invent them after a bad week to look rigorous
Why it works: Exactly. Before love. After-party, never, and un-hittable criteria are how zombies live.
- Write kill criteria before love.
- Name the evidence that stops it.
- Killing is not weakness.
- Do not write un-hittable criteria.
A team wants to A/B test a change to onboarding for a product used by a few hundred business accounts, where the outcome they care about is whether a team renews a year later.
What do you say?
The strongest answer: This is the wrong tool: too few accounts to separate a real difference from noise, and the outcome lands a year after the change. I would test it qualitatively and instrument earlier signals
Why it works: Right — small populations and long feedback loops are exactly where an experiment stops being able to answer the question.
- Experiments need volume and a short loop.
- Check the population first.
- Check when the outcome arrives.
- Otherwise, test another way.
Across eight teams, dozens of experiments ran last quarter. This quarter’s roadmap is almost identical to the one written before any of them reported.
What is broken, and what do you install?
The strongest answer: The decision step. Every test gets an owner, a decision written on the day it reports — ship, iterate or stop — and a place those decisions live so the next plan starts from them
Why it works: Right — experiments that report into a vacuum are theatre. Result, decision, owner, date, and a plan that is allowed to change.
- A test ends in a decision.
- Name the owner before it starts.
- Write ship, iterate or stop.
- Let the plan change.
How do you stop experiment theatre in the company?
The strongest answer: A primary-metric rule, a share of negative results, and no launch posts from peeks — I do not ban experiments
Why it works: Exactly. Rules and negative results. Bans, mandatory wins, and lift leaderboards are how you get fiction.
- Primary rule + share negatives.
- No peek-based launch posts.
- Do not ban experiments.
- Do not require a win or a lift board.