All jobs / Solutions / cloud architect
Reliability interview questions
Reliability questions a Solutions / cloud architect can be asked, from first job to senior. Open one to see the strongest answer and why it works.
How do you find single points of failure in a design you have just been handed?
The strongest answer: Walk the diagram asking “if this one thing dies, what stops?” — one instance, one zone, one database, one person with the password all count
Why it works: Exactly. Remove each element in turn, on paper. People and credentials are components too.
- Remove each box on paper.
- Ask what stops.
- Include zones, people and credentials.
- The cloud does not do this for you.
What is the difference between having backups and having disaster recovery?
The strongest answer: A backup is a copy of the data; disaster recovery is the tested ability to run the service again somewhere else within an agreed time — a copy nobody has restored is a hope
Why it works: Exactly. Copy versus rehearsed recovery. The restore test is what turns the first into the second.
- Backup: a copy.
- DR: rehearsed recovery in agreed time.
- Untested restores are hopes.
- Snapshots alone are not DR.
A service restarts in a loop every time its database has a slow minute.
What should a service’s health check actually test?
The strongest answer: Liveness: the process is running; readiness: it can serve traffic, including its critical dependencies — and keep them separate so a slow database does not cause a restart loop
Why it works: Exactly. Two questions, two probes, and failing readiness sheds traffic without killing the process.
- Liveness: process alive.
- Readiness: can serve, dependencies included.
- Keep them separate.
- Degrade on non-critical dependencies.
A downstream service is flaky. How should callers retry?
The strongest answer: A bounded number of retries with exponential backoff and jitter, only for idempotent operations or with an idempotency key, and a budget so a slow dependency cannot multiply load
Why it works: Exactly. Bounded, backed off, jittered, safe to repeat, budgeted.
- Bounded attempts.
- Exponential backoff with jitter.
- Only when safe to repeat.
- A retry budget protects the dependency.
Every time the recommendations service slows down, the product page exhausts its thread pool and goes down with it.
What does a circuit breaker add that timeouts and retries do not?
The strongest answer: After repeated failures it stops calling the dependency for a while and fails fast, giving it room to recover and protecting the caller’s threads; it then probes before fully reopening
Why it works: Exactly. Stop hammering, fail fast, probe, reopen. It protects both sides.
- Stop calling after repeated failure.
- Fail fast; free the caller’s resources.
- Probe, then reopen.
- Protects the dependency too.
Define RPO and RTO and explain who should set them.
The strongest answer: RPO is how much data loss is acceptable, measured in time; RTO is how long the service may be down; the business sets both and the architecture is costed against them
Why it works: Exactly. Business targets, technical cost. The architect connects the two.
- RPO: acceptable data loss.
- RTO: acceptable downtime.
- The business sets them.
- Architecture is costed against them.
A producer can generate far more work than the consumers can absorb. How does the design cope?
The strongest answer: Bound the queues, make producers slow down or shed when they fill, prioritise what matters, and make the lag visible — unbounded buffering only moves the failure
Why it works: Exactly. Bounded buffers, signalled slowdown, priority, visible lag.
- Bound the buffers.
- Producers slow or shed when full.
- Prioritise the work.
- Make lag visible.
Support tickets from small customers spike every time the largest customer runs its end-of-month batch.
One large tenant’s traffic is degrading everyone else on a shared platform. What are your options?
The strongest answer: Per-tenant rate limits and quotas first, then fair scheduling or dedicated pools for the heaviest tenants, priced accordingly — and metrics per tenant so you can see it next time
Why it works: Exactly. Limits, fairness, dedicated capacity where paid for, and per-tenant visibility.
- Per-tenant quotas and limits.
- Fair scheduling.
- Dedicated pools, priced.
- Per-tenant metrics.
The DR plan exists on paper. How do you find out whether it works?
The strongest answer: Run a tabletop walk-through to find missing steps, then a real restore into an isolated environment against the RTO and RPO, and schedule it so it stays true as the system changes
Why it works: Exactly. Talk it through, then actually do it, then repeat on a cadence.
- Tabletop first, for missing steps.
- Real restore in isolation.
- Measure against RTO and RPO.
- Repeat on a schedule.
Active-passive or active-active across regions — how do you choose?
The strongest answer: By the RTO the business will pay for and whether the data model tolerates concurrent writes; active-active buys fast failover at the price of conflict handling, so passive is often the honest choice
Why it works: Exactly. RTO and write conflicts decide it; the simpler option is often right.
- Start from the RTO the business funds.
- Can the data tolerate concurrent writes?
- Active-active means conflict handling.
- Passive is often the honest choice.
How should service-level objectives shape an architecture, rather than just measure it?
The strongest answer: An SLO tells you how much failure and latency budget each component may consume, so it drives redundancy, timeouts, caching and where to spend — a design without one is guessing
Why it works: Exactly. The SLO sets the budget the design spends.
- The SLO is the failure budget.
- It drives redundancy and timeouts.
- It says where to spend.
- Set it before designing.
After a two-hour regional outage, the executive team wants a global architecture by next quarter.
The CEO wants the platform “in every region so it never goes down”. What do you present?
The strongest answer: The actual availability and latency targets the business needs, the outage history, and the costed options from multi-zone through active-passive to active-active — with the data-conflict and operational burden of each named
Why it works: Exactly. Translate the wish into targets, then price the options honestly.
- Turn the wish into targets.
- Show outage history.
- Price zone, passive and active options.
- Name the operational burden.
How do you introduce failure testing in an organisation that has never done it?
The strongest answer: Start with game days on non-production against the failure modes the design claims to survive, fix what you find, then move to controlled experiments in production with a blast radius and a stop button — the goal is confidence, not surprise
Why it works: Exactly. Rehearsed, hypothesis-driven, bounded, and moving towards production as confidence grows.
- Game days first, non-production.
- Test the claims the design makes.
- Production with blast radius and stop button.
- Confidence, not surprise.
The board asks for disaster recovery to a second cloud provider. What do you tell them?
The strongest answer: That it is achievable only for a portable subset at high cost and constant rehearsal, that provider-wide outages are rarer than the failures already covered by multi-region, and to be sure the risk they fear is provider failure and not concentration or regulatory pressure
Why it works: Exactly. Name the real cost, question the risk, and answer the concern they actually have.
- Portable subset, high cost, constant rehearsal.
- Compare against multi-region coverage.
- Find the risk they actually mean.
- Answer that risk.