Streaming interview questions
Streaming questions a Data engineer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
A Kafka topic has six partitions and you start ten consumers in one consumer group. What happens?
The strongest answer: Each partition is read by exactly one consumer in the group, so six consumers get a partition each and four sit idle; more parallelism needs more partitions
Why it works: Exactly — partitions are the unit of parallelism within a group; consumers beyond that count wait.
- One partition, one consumer per group.
- Partitions cap group parallelism.
- Extra consumers idle.
- Other groups each get everything.
When is a nightly batch the honest choice?
The strongest answer: When the decision is daily and the source is already batch — I do not stream because the resume said “real time”
Why it works: Exactly. Freshness is a product requirement. Fashion is not. Batch still has late data.
- Match freshness to the decision.
- Batch is honest when the day is enough.
- Do not stream for the resume.
- Batch still restates.
A consumer had a bug for three days and wrote wrong rows. Can you get the events again from Kafka?
The strongest answer: Yes, if retention still covers those days: Kafka is a log and reading does not delete, so I reset the group’s offsets to before the bug and replay — after making the consumer idempotent so the rewrite does not double
Why it works: Right — the log plus offsets is what makes replay possible; retention is the limit and idempotence is the precondition.
- Kafka is a log; reads do not delete.
- Replay by resetting offsets.
- Retention bounds the replay.
- Make the consumer idempotent first.
A streaming aggregate needs a watermark. What are you trading?
The strongest answer: Completeness versus latency — a watermark closes a window and late events after that need a restatement path or they are dropped on purpose
Why it works: Exactly. A named trade. Zero watermark is “drop everything late.” Free correctness is a myth.
- Name the close time.
- Provide a restatement or accept drops.
- Watermarks are a trade.
- Zero ≠ perfect.
A Kafka consumer commits an offset as soon as it polls a record, then writes the record to the warehouse. The pod dies between the two. What happens?
The strongest answer: The record is lost — the committed position already moved past it. Commit after the write succeeds; a crash then re-reads the record, so the sink must de-duplicate or be idempotent
Why it works: Exactly — committing before the work is done is at-most-once by accident; commit after, and design the sink for duplicates.
- Commit after the write, not before.
- Commit early loses records.
- Commit late duplicates records.
- The sink must be idempotent.
Product wants “real-time revenue.” The decision is a daily price change. What do you offer?
The strongest answer: A freshness that matches the decision — hourly or daily — and I name the cost of true streaming; I do not build a toy stream for a slide
Why it works: Exactly. Freshness is a requirement. Logos and sub-second toys are slides. Personality dumps are the other extreme.
- Match freshness to the decision.
- Name streaming’s cost.
- No logo architecture.
- No personality weekly.
A vendor says the pipeline is “exactly once.” What do you ask?
The strongest answer: Where the dedupe key lives and what happens on a replay — the phrase is not a property of the logo
Why it works: Exactly. Keys and replay. Stickers and turning off upserts are how you get doubles.
- Ask for the dedupe key.
- Ask what replay does.
- The phrase is not enough.
- Keep idempotent writes.
Payment events are produced to Kafka and later some are missing downstream, though the producer logged no errors.
What producer settings do you check first?
The strongest answer: acks: with acks=1 only the leader confirms, so a leader failure before replication loses the record. I want acks=all with min.insync.replicas above one, the idempotent producer, and sends that wait for the result
Why it works: Right — durability is a producer-side decision: acknowledgement level, in-sync replicas, idempotence, and actually waiting for the ack.
- acks=all, min.insync.replicas above one.
- Idempotent producer for safe retries.
- Wait for the send result.
- Durability is a producer setting.
You need a 2-minute ops view and a 1-day finance book. How do you avoid two truths?
The strongest answer: One raw landing, two models with labels, and a documented restatement into the book — I do not run two unrelated ingest
Why it works: Exactly. One landing, two labelled products, a bridge. Dual ingest is how they drift.
- One landing.
- Two labelled models.
- Document the restatement.
- No dual unrelated ingest.
Producers emit JSON with no registry. What do you introduce first?
The strongest answer: A registry and compatibility rules on the topics that feed money — I do not boil every log line on day one
Why it works: Exactly. Start where breaks hurt. Bans, novels, and volunteers do not scale.
- Registry on money topics first.
- Compatibility rules.
- Do not boil the ocean.
- Do not volunteer-review random payloads.
Product wants “orders per five minutes”, “orders in the last hour, updated every minute” and “what a user did in one visit.” Which stream windows are those?
The strongest answer: Tumbling (fixed five-minute buckets, no overlap), sliding or hopping (an hour long, advancing every minute, so each event lands in many windows), and session (grouped by activity with a gap timeout, variable length)
Why it works: Exactly — and each has a cost: hopping windows multiply state, sessions need a gap and a late-event policy.
- Tumbling, sliding, session.
- Tumbling: fixed, no overlap.
- Sliding: overlapping, more state.
- Session: gap-defined, variable length.
When do you fund a streaming platform, not a job?
The strongest answer: When several products share the same low-latency facts and you can staff the ops — one toy job is not a platform
Why it works: Exactly. Shared need plus ops. Resume, morals, and demos are how you buy a lonely cluster.
- Shared low-latency facts.
- Staffed ops.
- One toy ≠ a platform.
- Do not buy a demo.