Big Data interview questions
Big Data questions a Data engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
A PySpark script runs ten transformations in a second, then a single count() takes twenty minutes. What is happening?
The strongest answer: Transformations are lazy — Spark only records a plan; count() is an action, which runs the whole plan. Nothing about the count itself was slow
Why it works: Exactly — this is the first thing to understand about Spark: transformations build a plan, actions execute it.
- Transformations plan; actions run.
- Nothing executes until an action.
- The action pays for the pipeline.
- Read the plan, not the clock.
A job works on a sample and dies with an out-of-memory error on the driver when run on the full table. What is the classic cause?
The strongest answer: Something pulls the whole dataset to the driver — collect(), toPandas() or a driver-side loop — which fits on a sample and not at scale. Keep the work distributed and inspect with take() or limit()
Why it works: Right — the driver is one machine; anything that materialises the data there stops scaling the moment the data grows.
- Never collect the whole dataset.
- The driver is one machine.
- Inspect with take or limit.
- Write results out, distributed.
A Spark job does a cross join “to be safe.” What happens?
The strongest answer: A blow-up — I filter and aggregate before I join, and I do not cross-join as a comfort blanket
Why it works: Exactly. Join after you shrink. Large clusters fund a bad plan.
- Filter and agg before join.
- No comfort cross-joins.
- Clusters do not fix combinatorics.
- Complete ≠ Cartesian.
One Spark task takes 50 minutes; the rest take 2. What do you look for?
The strongest answer: Skew — a hot key — then salt, pre-agg, or a separate path for the whale; I do not only add executors
Why it works: Exactly. One hot key. Executors fund the wait. Broadcasting the huge side is the other classic.
- Find the hot key.
- Salt, pre-agg, or split the whale.
- Executors last.
- Do not broadcast the big side.
Which operations make Spark shuffle data across the cluster, and why does it matter which they are?
The strongest answer: Wide transformations — groupBy, join, distinct, repartition, orderBy — need same-key rows on one node, so they shuffle over the network and cut a stage boundary; narrow ones like filter, select and map work within a partition
Why it works: Exactly — stages are cut at shuffles; knowing which operations are wide is how you read the Spark UI.
- Wide shuffles; narrow does not.
- groupBy, join, distinct, orderBy shuffle.
- Shuffles cut stages.
- Filter before the shuffle.
A join spilled to disk and died. What do you try before a 4× cluster?
The strongest answer: Broadcast the small side, pre-filter, or a better partition of the join key — I do not first buy machines
Why it works: Exactly. Shrink and broadcast. One partition and * caches are how you die louder.
- Broadcast the small side.
- Filter before the join.
- Mind the join-key partition.
- Machines last.
A colleague wraps a small Python function in a PySpark UDF to clean a column. The job slows down tenfold. Why?
The strongest answer: A plain Python UDF moves every row out of the JVM into a Python process and back, serialising as it goes, and the optimiser cannot see inside it; use built-in column functions, or a vectorised (pandas) UDF
Why it works: Right — row-by-row JVM-to-Python serialisation plus an opaque function; built-ins and Arrow-based UDFs avoid both costs.
- Built-ins first, then vectorised UDFs.
- Python UDFs serialise per row.
- The optimiser cannot see inside a UDF.
- Never collect to clean.
Five squads copy-paste Spark submit flags. What do you ship?
The strongest answer: A paved job template: runtime, defaults, and a cost ceiling — they add the transform, not a 12th spark-submit dialect
Why it works: Exactly. A template. Wikis, version zoos, and panic rewrites are how you pay five times.
- Ship a submit template.
- Squads add transforms.
- No version zoo.
- No panic engine swap.
A job ends by writing a table. Once it produces thousands of tiny files; another time four huge ones that take an hour on four tasks.
What controls this, and how do you set it?
The strongest answer: The partition count at write time. Coalesce reduces partitions without a shuffle when the output is small; repartition redistributes with a shuffle when a few tasks hold everything. I target hundreds of megabytes per file
Why it works: Exactly — output layout is a decision: coalesce down for small results, repartition when a few tasks hold everything.
- Partitions at write time decide files.
- Coalesce down without a shuffle.
- Repartition to fix skew.
- Target hundreds of megabytes per file.
Spark, a warehouse SQL, and a stream processor all exist. How do you pick?
The strongest answer: By job: heavy shuffle vs declarative models vs low-latency state — I do not pick from a conference talk
Why it works: Exactly. Job first. Resume, 5ms-SQL, and streaming the close are mismatches.
- Match the engine to the job.
- Do not Spark the world.
- Do not SQL the 5ms path.
- Do not stream the close for fashion.
A hobby Spark cluster has a fan club. How do you retire it?
The strongest answer: Show the SLO it never met, offer the paved runtime, and sunset with a date — I do not mock the fans in the all-hands
Why it works: Exactly. Replace and date. Mockery, tonight, and hidden bills make martyrs and shadows.
- Show the missed SLO.
- Offer the paved runtime.
- Sunset with a date.
- Do not mock or hide the bill.