Distributed Systems interview questions
Distributed Systems questions a Backend developer can be asked, from first job to senior. Open one to see the strongest answer and why it works.
The interviewer asks: “A client retries POST /payments because of a timeout. How do you not charge twice?” Strongest answer?
The strongest answer: Require an Idempotency-Key (or a natural unique like orderId). Store the first result and replay it for the same key. Timeouts then become safe.
Why it works: Exactly. That's the Stripe-style contract people expect in 2026. Unique constraint plus stored response beats hoping the network is nice.
- Clients retry; the server must be safe.
- Idempotency-Key or a natural unique.
- Store and replay the first response.
- Don't undo the first write on a retry.
The interviewer asks: “A downstream HTTP call timed out. Do you retry?” Strongest answer?
The strongest answer: Only if the call is idempotent (or has an idempotency key), with bounded exponential backoff and jitter. I don't retry 400s, and I don't hammer a 503 in a tight loop.
Why it works: Exactly. Retries are for transient faults on safe operations. Backoff + jitter protect the other side. 4xx is a contract error, not a blip.
- Retry transient + idempotent only.
- Backoff and jitter; cap the attempts.
- Don't retry 4xx.
- No infinite loops.
The interviewer asks: “Your HTTP client has no timeout. What's the risk?” Strongest answer?
The strongest answer: A hung dependency holds a thread/connection forever, the pool drains, and the whole API dies with it. I set connect and request timeouts on every outbound call — including the database.
Why it works: Exactly. Timeouts are blast-radius control. No timeout is an unbounded distributed deadlock.
- Every outbound call has a timeout.
- Hung deps shouldn't eat the pool.
- Timeouts are a feature, not noise.
- Don't convert them into fake 200s.
A RabbitMQ consumer acknowledges each message as soon as it arrives and then processes it. Messages go missing whenever a consumer pod restarts.
What do you change?
The strongest answer: Acknowledge after the work has committed rather than on receipt — an unacknowledged message is redelivered when the connection drops — and set a prefetch so one consumer does not hold more than it can finish
Why it works: Exactly — the ack is the consumer’s promise that the work is done, and prefetch keeps the redelivery set small.
- Ack after the work commits.
- Unacked messages are redelivered.
- Set prefetch to limit the blast.
- Keep the consumer idempotent.
The interviewer asks: “Your queue consumer ran twice for the same message. What do you do?” Strongest answer?
The strongest answer: Assume at-least-once. Make the handler idempotent — unique key, upsert, or a processed-message table — then ack. Don't assume the broker delivered once.
Why it works: Exactly. Duplicates are normal (timeouts, rebalances). Idempotency is the consumer's job.
- Queues are at-least-once by default.
- Handlers must be idempotent.
- Ack after success.
- Duplicates are a no-op, not a crash loop.
Two services each keep a copy of a customer’s address and sync with events. A product manager asks whether the two are “always the same”.
What do you tell them, and what does it force on your code?
The strongest answer: That the two converge once the events are delivered, with no promised time bound — so handlers must be idempotent and tolerate repeats and reordering, and product decides what a user sees during the window
Why it works: Exactly — convergence without a deadline. Naming the user-visible window is the part product actually needs from you.
- It converges, with no time bound.
- Make every handler idempotent.
- Expect repeats and reordering.
- Decide what users see meanwhile.
The interviewer asks: “The replica is behind. Do you serve stale reads or error?” Strongest answer?
The strongest answer: It depends on the use case: a feed can be stale; a balance after a debit cannot. I name the consistency the product needs (read-your-writes, bounded staleness) and pick primary vs replica per query, not as a global religion.
Why it works: Exactly. This is a per-operation choice. Stronger consistency costs latency and availability during partitions; weaker consistency needs a UX that can live with it.
- Staleness is a product choice per read.
- Money and “did my write stick?” → primary / RYW.
- Feeds can use replicas.
- Don't busy-wait on lag.
The interviewer asks: “How do you publish an event only if the order row actually committed?” Strongest answer?
The strongest answer: Transactional outbox: write the row and an outbox record in the same transaction; a publisher drains the outbox. No commit without the event, no event without the commit.
Why it works: Exactly. Dual-write (DB and broker as two commits) will desync. The outbox makes the event part of the same atomic unit; at-least-once drain + idempotent consumers finish the story.
- Outbox sits in the same DB transaction.
- A worker publishes after commit.
- Consumers stay idempotent.
- Don't dual-write and hope.
The interviewer asks: “We need “all services succeed or none” across order, inventory and billing. What do you reach for?” Strongest answer?
The strongest answer: A saga: a sequence of local transactions with compensations if a later step fails. Two-phase commit across microservices is rarely available and couples availability to every participant.
Why it works: Exactly. Sagas trade atomicity for explicit compensations and a state machine you can operate. That's the 2026 default across service boundaries.
- Across services: saga + compensation.
- 2PC is rarely the tool you have.
- Don't collapse services just to COMMIT.
- Every step has a failure story.
The interviewer asks: “Consumers are slow. The publisher is fast. What do you do?” Strongest answer?
The strongest answer: Apply backpressure: bound the queue, slow or reject producers, scale consumers, drop or shed load with a policy. Unbounded in-memory buffers turn a slow consumer into an OOM.
Why it works: Exactly. Backpressure is how healthy systems refuse work. A bound plus a 429/503 or a load-shed policy is explicit; an unbounded list is an accident.
- Bound the buffer; don't grow forever.
- Slow, reject, or shed with a policy.
- Don't stall HTTP on queue depth.
- Don't ack to hide the pile.
The interviewer asks: “A message fails every time. The consumer keeps retrying. What now?” Strongest answer?
The strongest answer: Bounded retries, then a dead-letter / poison queue with an alert. I don't block the partition on one bad payload, and I don't drop it silently.
Why it works: Exactly. Poison messages are a first-class ops path: isolate, alarm, replay after a fix. That's how you keep the rest of the topic flowing.
- Retry a little, then dead-letter.
- Alert on the poison pile.
- Don't ack failures.
- Don't stop the world for one payload.
Order-status events for the same order are arriving out of order at the consumer. The topic has twelve partitions.
Why, and what do you change?
The strongest answer: Ordering only holds within a partition and these events were spread across twelve — key them by order id, so every event for one order lands on the same partition and one consumer reads them in sequence
Why it works: Exactly — the key picks the partition and the partition is the ordering unit. It also sets your parallelism ceiling.
- Order holds inside a partition.
- Key by the entity id.
- Same key, same partition, same order.
- Partitions cap your parallelism.
The interviewer asks: “The user saves a profile then immediately GETs it from another region and sees the old one. Why?” Strongest answer?
The strongest answer: They read a replica that hasn't applied the write. For read-your-writes I stick to the primary (or a session that waits for the write's LSN). Cross-region lag is normal.
Why it works: Exactly. Replication is asynchronous unless you paid for sync. Name the consistency: RYW after a write, eventual for other readers.
- Async replicas lag; that's expected.
- Read-your-writes → primary or wait-for-LSN.
- Don't global-flush as a reflex.
- Don't serialise the world on the primary.
The interviewer asks: “A dependency is failing. How do you stop your service from dying with it?” Strongest answer?
The strongest answer: A circuit breaker (or concurrency limiter) plus a timeout: fail fast, shed or degrade, and probe half-open. I don't queue unbounded work behind a dead dependency.
Why it works: Exactly. Breakers protect your pool and give the dependency room to recover. Combine with a fallback the product agreed to.
- Timeout + breaker + fallback.
- Fail fast; don't queue forever.
- Don't retry storms.
- Don't paint 200 over an outage.
The interviewer asks: “The outbox publisher crashed after sending but before marking sent. What happens?” Strongest answer?
The strongest answer: It will send again. That's why consumers are idempotent and why the message carries a stable event id. I don't try to make the publisher exactly-once in-process.
Why it works: Exactly. At-least-once plus an event id is the robust design. Exactly-once is a property you approach with idempotent consumers, not a flag on the producer.
- Publisher duplicates are expected.
- Stable event id + idempotent consumer.
- Don't mark sent before send.
- Don't go back to in-memory dual-write.
The interviewer asks: “Recommendations are down. Checkout still needs to work. What do you do?” Strongest answer?
The strongest answer: Graceful degradation: serve checkout without recommendations, cache a last-known list, or hide the module. I don't let a non-critical dependency fail the money path — timeouts, breakers, and a product-agreed fallback.
Why it works: Exactly. Critical vs non-critical is an explicit list. Degrade the garnish; keep the transaction. That's resilience engineering, not a CSS trick.
- Classify critical vs optional deps.
- Fail optional; keep the money path.
- Timeouts and fallbacks, agreed with product.
- Don't buffer checkouts in RAM.
The interviewer asks: “We're adding a search index. Can the API write DB and Elasticsearch in one request?” Strongest answer?
The strongest answer: That's a dual write — one will fail. I commit to the database (and outbox) and let a consumer update search, or I use change-data-capture. I don't require both commits in the request.
Why it works: Exactly. Search is a derived view. The system of record commits first; search catches up. Idempotent upserts handle the lag.
- System of record first.
- Derive search via outbox or CDC.
- Don't dual-commit in the request.
- Don't 2PC the search engine.
Two app instances both believe they hold the lock for a nightly job. The lock is a key with a thirty-second expiry, and one instance was paused by a long garbage collection.
What happened, and what actually protects the write?
The strongest answer: The lease expired while the holder was paused and the second instance took it; the protection is a fencing token — an increasing number the resource checks, refusing any write that carries a stale one
Why it works: Exactly — a lock cannot stop a paused process. Only the resource rejecting a stale token protects the data.
- A lease can expire under you.
- Pauses break mutual exclusion.
- Fence writes with an increasing token.
- The resource must reject stale tokens.
The interviewer asks: “Active-active in two regions — what's the hard part?” Strongest answer?
The strongest answer: Data: conflict rules, latency, and what happens when the regions split. I choose a truth (single primary, CRDTs, or “last write with a clock I don't trust”) and I test the split. DNS and stateless pods are the easy half.
Why it works: Exactly. Multi-region is a data problem wearing a compute costume. Failover you haven't split-brained in a game day is a rumour.
- Multi-region is a data-consistency design.
- State the conflict rule; test the split.
- Pods + DNS aren't the hard part.
- Don't trust wall clocks for LWW.