Data Analysis interview questions
Data Analysis questions a Data analyst / BI can be asked, from first job to senior. Open one to see the strongest answer and why it works.
Average order value is well above what most customers actually spend. Which number goes on the slide?
The strongest answer: Both, labelled — the median for what a typical order looks like and the mean beside it, because a handful of very large orders pull the average up, and I show the distribution
Why it works: Exactly — say which statistic you are using and why. The mean is still the right one when the total matters.
- Name the statistic and the reason.
- Skew pulls the mean.
- Median for typical, mean for totals.
- Show the distribution, not one number.
Region A is 10% and Region B is 20%. Is the company 15%?
The strongest answer: Only if the regions are the same size — otherwise I go back to the raw counts; I do not average averages and call it the company
Why it works: Exactly. Weighted by the denominator. Unweighted means are a slide lie.
- Weight by the denominator.
- Go back to counts.
- Equal-size is the only time 15% is free.
- Do not pick the prettier percent.
A PM says “we will A/B it and see.” What do you require first?
The strongest answer: A hypothesis, a primary metric, and who is in the experiment — I do not run a tour of twenty metrics and pick a winner after
Why it works: Exactly. Design before peeking. Invented power and one-user holdouts are theatre.
- Hypothesis + primary metric + population.
- Do not shop twenty metrics after.
- Do not invent n.
- Do not promise p-values.
You surveyed 80 users and want to say “our users think.” What do you write?
The strongest answer: These 80, how we found them, and what we cannot say about everyone — I do not dress a convenience sample as the population
Why it works: Exactly. A sample is a sample. Hidden n and “statistically everyone” are honesty misses.
- Name n and how you found them.
- Name what you cannot say.
- Do not promote a convenience sample.
- Do not hide n.
A PM refreshes the A/B dashboard daily and wants to ship on the first green day. What do you say?
The strongest answer: Peeking inflates false wins — we ship on the design we wrote (time or events), or we use a method that allows sequential looks; we do not dance for the first star
Why it works: Exactly. The design includes when you look. Invented p-values are a red flag.
- Look when the design says.
- Or use a sequential method.
- Do not ship the first star.
- Do not invent a calmer p.
Two support teams have the same average resolution time, and the ops lead concludes they are performing identically.
What do you check before you agree?
The strongest answer: The spread — I compare the two distributions and a high percentile such as p90, because one team can be steady while the other mixes very fast tickets with a tail of very slow ones
Why it works: Right — customers feel the tail. Two identical means can hide completely different experiences.
- Averages hide the spread.
- Compare distributions, not centres.
- Report a high percentile too.
- The tail is what customers feel.
A “week-0 conversion” includes events from week 3 because the join used user_id only. What is that?
The strongest answer: Leakage — future facts in an early bucket — I join on user and time, or I will flatter the funnel
Why it works: Exactly. Time belongs in the join. “More complete” is how you invent a miracle.
- Join user + time.
- No future facts in early buckets.
- Keep timestamps.
- DISTINCT is not a time join.
It is the 10th of the month, and the monthly revenue chart shows this month far below the last one.
What do you say?
The strongest answer: That the month is only a third gone — I compare the first ten days with the same ten days of earlier months, and mark the current bar on the chart as incomplete
Why it works: Exactly — like-for-like windows, and a chart that admits which bar is still filling.
- Never compare a part to a whole.
- Match the window exactly.
- Mark the incomplete period.
- Do not extrapolate to fill it.
You need a conversion rate across countries. Do you average country rates?
The strongest answer: No — I take sum(converted) / sum(users), or I show country rates separately; an unweighted mean of rates is not the company
Why it works: Exactly. Ratio of sums, or a distribution. Dropping small and calling a median “global” are different products — name them.
- Ratio of sums for the company.
- Or show the distribution.
- Do not unweight by habit.
- Name it if you drop small.
The overall A/B is flat; one segment is “up.” What do you write?
The strongest answer: That the primary is flat, that the segment was not pre-registered, and that we will not ship a story as a finding
Why it works: Exactly. Exploratory is exploratory. Backdated registries and p-hacking are red flags.
- Lead with the primary.
- Label unplanned segments.
- Do not backdate a registry.
- Do not shop p-values.
You randomise users but analyse page views. What is wrong?
The strongest answer: The unit of analysis must match the unit of randomisation, or you pretend views are independent and you overstate the win
Why it works: Exactly. Views inside a user are clustered. More rows are not more independent experiments.
- Analyse the unit you randomised.
- Views are clustered in users.
- More rows ≠ more experiments.
- Do not cluster only when it hurts.
How do you keep an experimentation program honest when people want wins?
The strongest answer: Pre-registration, a primary, guardrails, and a place that publishes flats — I do not fund only the tests that “worked”
Why it works: Exactly. Flats are data. Trophy walls and must-win rules are how you p-hack a culture.
- Pre-register the primary.
- Publish flats.
- Guardrail metrics.
- No must-win rule.
The average treatment effect is flat; you suspect it helped new users and hurt old ones. What is honest?
The strongest answer: A pre-specified heterogeneity plan, or a labelled exploration with a follow-up test — I do not ship a split we discovered in the leftovers
Why it works: Exactly. Heterogeneity is a design or a next test. Leftover shipping and dogma averages are both lazy.
- Pre-specify or follow up.
- Label exploration.
- Do not ship leftovers.
- Averages can hide harm — say so.
A correlational chart is titled “how onboarding causes retention.” What do you change?
The strongest answer: The verb — associated, among users who, after we launched — unless we designed a causal study I will not write “cause”
Why it works: Exactly. Verbs are claims. p-values do not mint causes. Whispered footnotes lose to titles.
- Match the verb to the design.
- No “cause” without a causal design.
- p ≠ cause.
- Do not hide the truth in a footnote.
NPS is 72 from people who finished a 20-minute survey. What do you refuse to say?
The strongest answer: That “customers love us” as a population claim — I will say who answered and who we never heard
Why it works: Exactly. Respondents are a segment. Invented quartiles and mind-reading non-respondents are honesty misses.
- Name who answered.
- Name who did not.
- No invented quartiles.
- No mind-read of non-respondents.
A sign-up funnel shows most of the loss at step three, and the PM wants to redesign that screen this sprint.
What do you do before that work starts?
The strongest answer: Check the funnel is real — are the events ordered per user inside a window, is step three skippable, did its event stop firing on some clients? I trace a sample of real users through it
Why it works: Right — a missing event and a real drop look identical in a funnel chart. Instrumentation first, then the screen.
- Verify the funnel before the finding.
- Check event order and window.
- Ask whether the step is required.
- Trace a sample of real users.
You cannot randomise users (a marketplace). What do you not pretend?
The strongest answer: That a simple user A/B still applies — I will talk switchback, geo, or holdout designs and their weaker claims; I will not fake independence
Why it works: Exactly. Design follows interference. Textbook p-values on interfered units are costume.
- Name the interference.
- Pick a design that fits.
- Weaken the claim if you must.
- Do not costume a user p-value.
A test would hide prices from one city for a month. What do you raise besides power?
The strongest answer: Harm, consent, and who can stop it — a significant result does not make a cruel design honest
Why it works: Exactly. Design has ethics. Faster or renamed is not a review. Significance is not a blessing.
- Raise harm and stop-rules.
- Significance ≠ blessing.
- Do not rename the city as ethics.
- Someone must be able to stop it.
One enterprise customer is 40% of revenue. How do you show growth?
The strongest answer: With and without them, named — I do not drop them as an “outlier” so the SMB story looks clean
Why it works: Exactly. Concentration is the story. Silent winsorizing is a second book.
- Show with and without.
- Name the whale.
- Do not silent-winsorize.
- Median is a different claim.
A “retention” chart mixes users who could have returned with users who just signed up yesterday. What do you fix?
The strongest answer: The eligibility window — only people who had time to come back belong in that day’s denominator — I do not call a half-open cohort “retained”
Why it works: Exactly. Retention is a follow-up clock. Half-open denominators flatter the present.
- Eligible-only denominators.
- Give people time to return.
- Do not flatter with half-open cohorts.
- Do not drop yesterday in silence.
Overall conversion fell last quarter, yet every single channel’s conversion rate went up.
How do you explain that to the exec team?
The strongest answer: The mix moved — more traffic came from the weakest-converting channel, so the total fell while every part rose; I show the rates beside the volumes and split the change into a rate effect and a mix effect
Why it works: Exactly — naming it is not enough; the decomposition tells them which half to act on.
- Rates and weights move separately.
- Show volumes next to rates.
- Split rate effect from mix effect.
- Say which half to act on.
What does an experimentation platform owe a PM on Tuesday?
The strongest answer: A design they cannot silently peek off of, a readout that leads with the primary, and a way to stop — not a raw SQL dump
Why it works: Exactly. The road makes honesty the easy path. Winner-buttons and hidden flats are the product you must not ship.
- Hard-to-peek designs.
- Primary-first readouts.
- A stop path.
- No hidden-flat mode.
When do you staff causal work versus descriptive?
The strongest answer: When the decision is a change we can assign or a design we can defend — I do not put “causal” on every correlation to sound senior
Why it works: Exactly. Method follows the decision. Status and buttons are not designs.
- Causal when you can assign or defend a design.
- Description is not junior.
- No vendor-button causes.
- Do not label every r as causal.
Shipping without a test is how the company moves. What do you change first?
The strongest answer: Put holdout rate and “decided without evidence” next to delivery metrics — then staff the paved experiment path — do not start with a shame list
Why it works: Exactly. Visibility plus a road. Shame, freezes, and hidden logs grow shadows.
- Publish holdout and blind-ship rate.
- Fund the paved path.
- No shame list.
- No hidden book.
How do you stop the company from running random tests?
The strongest answer: A learning agenda: the few questions this quarter, mapped to designs — curiosity is welcome in exploration, not as a production traffic lottery
Why it works: Exactly. Scarce traffic, scarce questions. Bans and lotteries and endless backlogs are the failure modes.
- Few questions this quarter.
- Map each to a design.
- Exploration ≠ prod lottery.
- No 200-item museum.
Main is full of green A/B badges and nobody trusts them. What do you inspect?
The strongest answer: Peeking, optional primaries, hidden flats, and who can ship without a readout — cheap green is the expensive kind
Why it works: Exactly. Distrust is a clue the program is decorative. More tests on a lying board make a larger lie.
- Inspect peeking and hidden flats.
- Inspect bypasses.
- Do not paint a brighter badge.
- Do not hide distrust.