Data Lakes interview questions
Data Lakes questions a Data engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
Parquet or Avro — when do you use each in a pipeline?
The strongest answer: Parquet is columnar, compressed and splittable — for analytics at rest where queries read a few columns of many rows; Avro is row-based with the schema embedded — for record-at-a-time writes and event transport
Why it works: Right — column-oriented for reading slices of many rows, row-oriented for writing and shipping whole records; most lakes use Avro in flight and Parquet at rest.
- Columnar at rest, row-based in flight.
- Parquet: few columns of many rows.
- Avro: whole records, schema embedded.
- CSV is not a storage format.
What is different about writing a table to S3 or GCS compared with writing to a local filesystem?
The strongest answer: Objects are immutable with no real directories: no in-place edit, a rename is copy plus delete and not atomic, and listing is a request — so lake writers commit through a transaction log or manifest
Why it works: Exactly — no in-place edit, no atomic rename, listing as an API call; table formats exist to make a multi-file write look atomic.
- Objects are immutable; no atomic rename.
- No directories, only prefixes.
- Listing is a request with a cost.
- Commit through a log or manifest.
A streaming job wrote 200,000 tiny files for one day. What is the smell?
The strongest answer: Small-file tax — compact, or write larger rolls; I do not celebrate file count as throughput
Why it works: Exactly. Listing and opening files is the bill. File count is not a KPI.
- Compact tiny files.
- Write larger rolls.
- File count ≠ throughput.
- Listing is part of the cost.
Two Parquet datasets hold the same rows. A filter on order_date reads a tenth of one and nearly all of the other. Why?
The strongest answer: Parquet keeps min/max statistics per row group; data written sorted or partitioned by order_date gives each row group a narrow range the reader can skip, while random order makes every row group cover every date
Why it works: Right — predicate pushdown works through row-group statistics; layout (sort or partition by the filter column) is what makes the statistics selective.
- Layout makes statistics selective.
- Row groups carry min/max per column.
- Sort or partition by the filter column.
- Check rows read, not just seconds.
Time travel is on and storage doubled. What do you set?
The strongest answer: A retention that matches how far you actually restate — I do not keep “forever, just in case” and I do not vacuum yesterday if last week’s backfill still needs it
Why it works: Exactly. Retention is a product of restatement. Forever and zero are ideologies.
- Retention = restatement window.
- Not forever.
- Do not vacuum inside that window.
- Not zero as a personality.
Bronze / silver / gold is on the slide. What makes it real?
The strongest answer: Contracts and tests at each hop, and a ban on gold reading bronze “just this once” — names without gates are folders
Why it works: Exactly. Gates, not colours. Late raw-as-gold and extra metals are fashion.
- Tests at each hop.
- Gold does not read bronze.
- Names ≠ gates.
- Do not invent metals.
BI wants to query Parquet files in the lake. External table over the files, or load them into the warehouse?
The strongest answer: External for exploration and rarely read data, since there is no copy to keep in sync; load, or use a managed lakehouse table, for dashboards hit all day, where native layout, statistics and caching pay for the copy
Why it works: Exactly — decide by access pattern: a hot dashboard wants native layout and statistics; the long tail wants no second copy.
- Hot: load or manage; cold: external.
- External avoids a second copy.
- Native tables bring layout and stats.
- Never point BI at raw files.
What must stay true so you can rebuild gold from 90 days ago?
The strongest answer: Raw landings, the model versions, and the seeds — I do not overwrite raw with “cleaned” and call it bronze
Why it works: Exactly. Replay needs raw and code. Latest-only and memory are how you cannot rebuild.
- Keep raw immutable.
- Version the models.
- Keep seeds.
- Do not clean in place on bronze.
Why bother with a table format (Iceberg/Delta/Hudi) instead of a pile of Parquet?
The strongest answer: ACID, time travel, and partition evolution you can operate — a pile has no transaction and no safe rewrite
Why it works: Exactly. Operable tables. Keynotes, skipped tests, and no-orchestrator are not reasons.
- Transactions and safe rewrites.
- Time travel you can retain.
- Not a keynote.
- Not a skip-tests card.
How do you implement erasure in a lake that time-travels?
The strongest answer: A delete that reaches current and the retained history you still expose, plus backups per policy — I do not say “it’s immutable so we cannot”
Why it works: Exactly. Policy reaches copies you can still query. Immutable-as-excuse and useful-forever history of a person are how you fail a review.
- Delete current and retained history you expose.
- Follow backup policy.
- Immutable ≠ we cannot.
- Keep the audit of the delete.
Streaming writes leave a trail of tiny files. What do you schedule?
The strongest answer: A compaction you can bound — after a watermark or on a clock — so reads stay cheap; I do not compact in the write path if that is how we miss the SLA
Why it works: Exactly. Compact as a job with a bound. Magic, blocking every batch, and weekly drops are how you miss or lose.
- Schedule bounded compaction.
- Do not block every ingest.
- Lakes do not self-heal files.
- Do not DROP as maintenance.
Spark, a SQL engine and the BI tool each keep their own idea of which tables exist in the lake, and they disagree about columns.
What do you standardise first?
The strongest answer: One catalog every engine reads and writes through — the single registry of tables, schemas, locations and permissions — with table formats that carry their own schema; engines that cannot use it lose write access
Why it works: Right — the catalog is the lake’s source of truth; without it every engine is a private dialect of what the data is.
- One catalog, every engine.
- Tables, schemas, locations, permissions.
- Table formats carry the schema.
- No write access outside the catalog.
After a merger you have two lakes. How do you close it?
The strongest answer: A written primary, a dual-run of certified gold, and a sunset — I do not UNION ALL the people tables and I do not pick in a hallway
Why it works: Exactly. One paper, dual-run, sunset. UNION ALL of people, forever-both, and unmapped thirds are how the merger never ends.
- Write the primary.
- Dual-run certified gold.
- Sunset the rest.
- Do not UNION ALL people.