DevOps / SRE / cloud interview questions

A founder says “we are in the cloud, so we are highly available.” What do you reply?

The strongest answer: A region and an AZ are not a design — HA is multi-instance, multi-AZ, and a tested failover, not a billing account

Why it works: Exactly. The cloud sells building blocks. HA is how you place them.

• Name instances and AZs.
• Test failover.
• Do not equate a bill with HA.
• Snapshots ≠ high availability.

A teammate wants to “save CI minutes” by running tests only before release. What do you say?

The strongest answer: Run the fast tests on every push — a broken main is more expensive than the minutes — and keep a longer suite on a schedule or a label

Why it works: Exactly. Fast signal on every change. Long suites are a different knob.

• Fast tests on every push.
• Longer suite on a slower train.
• Do not wait for release day.
• Do not make the gate unbearable.

`./deploy.sh` says “Permission denied” and `ls -l` shows `-rw-r--r--`. What do you do?

The strongest answer: Add the execute bit for the owner — `chmod u+x deploy.sh` (or 755 if others run it) — and commit the mode so the pipeline gets the same file

Why it works: Right — r, w and x are separate bits, and the file needs x to be run directly. Fixing the mode fixes it for everyone who checks it out.

• Read the mode before you change it.
• r, w, x — owner, group, others.
• Add only the bit that is missing.
• Never reach for 777.

def add(item, bag=[]): bag.append(item); return bag — you call add(1) then add(2). What comes back the second time?

The strongest answer: [1, 2] — the default list is created once when the function is defined, so every call without a bag shares it; use bag=None and build a new list inside

Why it works: Exactly. One default object, shared for the life of the function. None plus a fresh list inside is the fix.

• Default arguments are evaluated once, at definition.
• A mutable default is shared by every call.
• Use None and create the value inside the function.
• The bug is silent, not an error.

What is the difference between a region and an availability zone?

The strongest answer: A region is a geographic area with several availability zones; each zone is one or more separate data centres with its own power and network, so spreading across zones survives a data-centre failure

Why it works: Right. The zone is the fault boundary you design around; the region is where your data lives.

• Region: a geography with multiple zones.
• Zone: isolated power, cooling and network.
• Spread across zones for availability.
• Spread across regions for disaster recovery.

An interviewer says: “Design a URL shortener.” What do you do first?

The strongest answer: Ask what it must do and at what scale — reads vs writes, expiry, custom aliases, analytics — and state the assumptions you will design against before drawing anything

Why it works: Right. Requirements and numbers first; the boxes follow from them.

• Clarify functional and non-functional requirements first.
• State assumed scale in numbers.
• Only then draw.
• Check assumptions with the interviewer.

A service is “down.” Where do you look first?

The strongest answer: Whether the name resolves, whether I can reach the port, then the app — I do not start in the application logs if the packet never arrived

Why it works: Exactly. DNS, reachability, then the app. Layer the investigation.

• Resolve the name.
• Reach the port.
• Then read the app.
• Do not reboot the world first.

What is the difference between an image and a container?

The strongest answer: An image is the immutable recipe; a container is a running instance of that recipe — I do not “SSH into the image to fix prod”

Why it works: Exactly. Recipe versus process. You ship images; you run containers.

• Image = immutable artifact.
• Container = running instance.
• Fix the image, do not pet the container.
• Do not call it a small VM and stop thinking.

What does a load balancer do, and what does it not do?

The strongest answer: Spreads requests across healthy instances and drops unhealthy ones from rotation; it does not make a slow service fast or a stateful one safe to scale

Why it works: Exactly. Distribution and health checks. It hides instance failure, not design failure.

• Distributes across healthy instances.
• Health checks remove the sick ones.
• Does not fix slow code or shared state.
• Not an availability guarantee on its own.

When would you choose a tuple over a list?

The strongest answer: When the collection is fixed and its position means something — a coordinate, a record, a dict key — since tuples are immutable and therefore hashable

Why it works: Right. Immutability is the point, and hashability is what it buys you.

• Tuple = fixed, positional, hashable.
• List = a sequence you intend to change.
• Only a tuple can key a dict or join a set.
• Choose on meaning, not on length.

IaaS, PaaS, SaaS — what does each one leave to you?

The strongest answer: IaaS gives you machines and networks and you run everything on them; PaaS runs the platform and you bring code and data; SaaS is a finished application you configure

Why it works: Exactly. The line that moves is who patches, scales and operates the layer underneath.

• IaaS: VMs, disks, networks — you run the rest.
• PaaS: bring code and data, the platform runs it.
• SaaS: a finished product you configure.
• The difference is who operates each layer.

A `.env` with production URLs and a shared password is in git “for convenience.” What do you do?

The strongest answer: Remove it, rotate the password, and keep an `.env.example` with fake values — convenience is not a secret store

Why it works: Exactly. Example file, real secrets in the vault, rotate what was committed.

• Vault the real values.
• Commit only examples.
• Rotate what was in git.
• Do not encrypt-with-README.

A service restarts in a loop every time its database has a slow minute.

What should a service’s health check actually test?

The strongest answer: Liveness: the process is running; readiness: it can serve traffic, including its critical dependencies — and keep them separate so a slow database does not cause a restart loop

Why it works: Exactly. Two questions, two probes, and failing readiness sheds traffic without killing the process.

• Liveness: process alive.
• Readiness: can serve, dependencies included.
• Keep them separate.
• Degrade on non-critical dependencies.

A service fails to start with “address already in use” on port 8080. How do you find out what holds it?

The strongest answer: `ss -ltnp` (or `lsof -i :8080`) shows the listening socket and the owning PID; then I decide whether it is a stale copy of the same service or something that belongs there

Why it works: Right — the socket table tells you exactly who listens; guessing from process names does not.

• Ask the socket table, not the process list.
• ss -ltnp or lsof -i :port.
• Identify before you kill.
• Do not reboot to avoid looking.

You read a 40 GB log file to count matching lines. What shape does the code take?

The strongest answer: Iterate the file object line by line, or write a generator that yields matches — one line is in memory at a time, so the file size stops mattering

Why it works: Exactly. Streaming turns memory cost into a constant.

• Iterating a file yields lines lazily.
• Generators keep memory constant.
• readlines and read materialise everything.
• yield lets you compose stages without buffering.

A user has an allow policy for an action and their group has a deny for it. What happens?

The strongest answer: The deny wins — policy evaluation starts from implicit deny, an explicit allow grants, and an explicit deny anywhere overrides every allow

Why it works: Right. Deny beats allow, always; that is what makes guardrail denies trustworthy.

• Default: implicit deny.
• Explicit allow grants.
• Explicit deny overrides everything.
• Guardrails are written as denies for that reason.

URL shortener: how do you generate the short code?

The strongest answer: Either hash the long URL and take the first characters, handling collisions by retry, or hand out pre-generated unique codes from a counter or key service; say which, and why — a counter gives guaranteed uniqueness but predictable codes, a hash is stateless but needs collision checks

Why it works: Right. Name the scheme and its trade-off.

• Hash-and-truncate or pre-generated keys.
• Handle collisions explicitly.
• Predictability vs statelessness.
• Length from the target count.

What belongs in a CI/CD pipeline before production?

The strongest answer: Build a unique artifact, test it, scan what you reasonably can, and deploy that same artifact — I do not rebuild on the way to prod

Why it works: Exactly. One artifact, promoted. Rebuilds on the path are new unknowns.

• Build once.
• Test that artifact.
• Promote it.
• Do not rebuild for prod.

What is a decorator, really?

The strongest answer: A function that takes a function and returns a replacement — @log above def f is just f = log(f), which is why the wrapper should carry functools.wraps

Why it works: Right, and wraps is the part people leave out until a name or docstring goes missing.

• @d over f means f = d(f).
• The decorator runs at definition time.
• functools.wraps keeps __name__ and __doc__.
• A decorator with arguments is one more layer.

A service on a VM needs to read from a storage bucket. How does it get permission?

The strongest answer: Attach a role to the instance with a policy scoped to that bucket; the SDK picks up temporary credentials from the instance metadata, and nobody ever creates or stores an access key

Why it works: Exactly. Instance roles are the pattern; keys on disks are the breach.

• Instance or workload role, scoped to the resource.
• Temporary credentials from metadata.
• No access keys on machines.
• Least privilege on the policy.

Design a rate limiter: which algorithm and where does its state live?

The strongest answer: Token bucket or sliding window per client key, with the counters in a fast shared store like Redis so every gateway instance sees the same count; say what happens when the store is down — fail open or closed — because that is the real design decision

Why it works: Right. Shared, fast state, and a stated failure mode.

• Token bucket or sliding window.
• Shared fast store for counts.
• Keyed per client.
• Decide fail-open vs fail-closed.

A runbook installs a CLI on every server with `curl https://… | sudo bash`. What do you prefer, and why?

The strongest answer: A package from a trusted repository, or a pinned release whose checksum or signature we verify, baked into the image — reviewable, reproducible and upgradable

Why it works: Right — package management gives you versions, verification and an upgrade path; a pipe to root gives you whatever the server sent today.

• Install from a repo or a verified pin.
• Pin the version.
• Verify checksum or signature.
• Bake it into the image.

CI rebuilds a 12-minute Docker image on every commit. What do you change first?

The strongest answer: Order the Dockerfile so dependency layers stay cached, and cache those layers in CI — I do not start with a bigger runner

Why it works: Exactly. Cache-friendly layers, then CI cache. Hardware is later.

• Stable layers first in the Dockerfile.
• CI cache on those layers.
• Do not git the image.
• Do not skip to a bigger runner.

You are designing the address space for a company’s first cloud networks. What do you decide up front?

The strongest answer: Non-overlapping private ranges for every network, environment and the on-premises estate, room to grow, subnets split per zone and tier, and a recorded allocation plan

Why it works: Right — overlapping ranges cannot be peered or routed later, and re-addressing a live network is a painful project.

• Non-overlapping, generous, recorded.
• One range per network and environment.
• Include the on-premises ranges.
• Subnets per zone and tier.

Twelve services, no tracing. How do you introduce distributed tracing without a year-long programme?

The strongest answer: Standardise the context propagation and instrumentation library first, instrument the gateway and the two slowest paths, sample sensibly, and let each team adopt the shared setup on its own schedule

Why it works: Exactly. Standard first, prove value on the painful paths, then spread.

• Standardise propagation first.
• Instrument the slowest paths.
• Sample deliberately.
• Spread by paved path, not decree.

What does zero trust change about how services talk to each other inside the network?

The strongest answer: Being on the network grants nothing: every call authenticates the caller, is authorised for the action, and is encrypted, so a compromised host cannot reach everything else

Why it works: Exactly. Identity per call, authorisation per action, encryption in transit — inside as well as outside.

• Network location grants nothing.
• Authenticate every caller.
• Authorise every action.
• Encrypt inside as well as outside.

A CPU-bound function is too slow. You add threads and nothing improves. Why?

The strongest answer: The GIL lets only one thread execute Python bytecode at a time, so threads help blocking I/O, not computation — use processes, a native library that releases the GIL, or move the work out of Python

Why it works: Exactly. Threads for waiting, processes for computing.

• The GIL serialises bytecode execution.
• Threads still win on I/O waits.
• multiprocessing for CPU-bound work.
• NumPy and friends release the GIL in C.

The business asks for a disaster recovery plan. What two numbers do you ask for first, and why?

The strongest answer: RPO — how much data loss is acceptable — and RTO — how long the service may be down; together they decide whether backups, a warm standby or an active second region is the right answer, and what it costs

Why it works: Right. Every DR design is a point on the RPO/RTO/cost triangle; get the two numbers before the architecture.

• RPO: acceptable data loss.
• RTO: acceptable downtime.
• They pick backup, warm standby or active-active.
• Cost rises steeply as both approach zero.

Your queue delivers at least once. What does that force on every consumer?

The strongest answer: Idempotency: each message carries a key and the consumer records what it has processed, so a redelivery after a crash or a slow ack is a no-op; you design the side effects to tolerate repeats rather than hoping for exactly-once

Why it works: Right. Repeats are certain; make them harmless.

• At-least-once means duplicates.
• Consumer keeps processed keys.
• Side effects tolerate repeats.
• Do not rely on exactly-once.

The team wants true continuous deploy to production. What do you still insist on?

The strongest answer: An automated gate that matches the risk (tests, a canary, a halt) and an owner who can stop the train — CD is not “no judgement”

Why it works: Exactly. Automate the judgement you already trust. CAB-on-every-commit is the opposite failure.

• Automate the gate.
• Keep a halt.
• Do not CAB every commit.
• Do not drop signing.

The pods are Running, but calls to the Service get connection refused.

Walk me through the diagnosis.

The strongest answer: `kubectl get endpoints` for the Service — empty means the selector does not match the pod labels or the pods are not Ready; populated means check targetPort against the container port and the readiness probe

Why it works: Exactly — a Service routes to endpoints, and endpoints are pods matching the selector and passing readiness. Empty list, wrong labels or not ready.

• Service routes to endpoints.
• Check kubectl get endpoints.
• Selector, readiness, targetPort.
• Running is not Ready.

One synchronous database call inside an async handler makes the whole service slow. Why?

The strongest answer: A blocking call stops the event loop, so no other coroutine can run until it returns — use an async driver, or push it to a thread with asyncio.to_thread

Why it works: Right. One blocking call in an async service is an outage waiting for load.

• Blocking code freezes the whole event loop.
• Use async drivers end to end.
• asyncio.to_thread for unavoidable blocking calls.
• One offender starves every other coroutine.

Active-passive versus active-active across regions — what does active-active cost you beyond money?

The strongest answer: Data consistency: two regions taking writes means conflict resolution, replication lag and a data model that tolerates both, so active-active is a design constraint on the application, not only a bigger bill

Why it works: Right. The hard part is the data; the compute is the easy part.

• Active-active means writes in two places.
• Conflict resolution and lag are yours.
• Design the data model for it.
• Active-passive keeps one truth.

A hot cache key expires and 5,000 requests miss at once. Design against it.

The strongest answer: Coalesce: the first miss takes a lock or a single-flight slot and rebuilds while the others wait or serve the stale value; add early refresh before expiry with jittered TTLs so keys do not expire together; the database sees one read, not thousands

Why it works: Right. Single flight, stale-while-revalidate, jitter.

• One rebuild, others wait or serve stale.
• Refresh before expiry.
• Jitter TTLs.
• DB sees one read.

The RTO agreed with the business is four hours; the next test restore from the archive tier is scheduled for next week.

Backups were moved to the cheapest archive storage tier and the bill fell. What did you just change about disaster recovery?

The strongest answer: Archive tiers take hours to retrieve and charge for retrieval, so the recovery time objective may no longer be achievable — keep recent backups in a fast tier, archive only the older ones, and test a restore from the archive

Why it works: Right — a storage tier trades retrieval time and cost against price; the RTO must still hold.

• Tiers trade retrieval time for price.
• Recent backups stay fast.
• Archive the old ones.
• Test a restore against the RTO.

Three independent API calls run one after another in an async function. How do you overlap them?

The strongest answer: await asyncio.gather(a(), b(), c()) so all three are in flight at once — and decide what return_exceptions should do, because by default the first failure propagates

Why it works: Exactly, and the exception policy is the part usually left unconsidered.

• gather runs awaitables concurrently.
• Sequential awaits are sequential.
• return_exceptions decides fail-fast or collect.
• TaskGroup gives structured cancellation.

After a two-hour regional outage, the executive team wants a global architecture by next quarter.

The CEO wants the platform “in every region so it never goes down”. What do you present?

The strongest answer: The actual availability and latency targets the business needs, the outage history, and the costed options from multi-zone through active-passive to active-active — with the data-conflict and operational burden of each named

Why it works: Exactly. Translate the wish into targets, then price the options honestly.

• Turn the wish into targets.
• Show outage history.
• Price zone, passive and active options.
• Name the operational burden.

The same class of incident keeps recurring across different teams. What does the architect do?

The strongest answer: Read the reviews together to find the shared cause — a missing platform capability, a default, a pattern — fix it once at the platform or standard level, and track whether the class disappears

Why it works: Exactly. Systemic causes need systemic fixes, then evidence that they worked.

• Find the shared cause across reviews.
• Fix at the platform or standard level.
• Prefer defaults over documents.
• Track whether the class disappears.

You are designing a Python service that fans out to many slow downstreams. What shapes the design?

The strongest answer: Async end to end so waiting costs a task rather than a thread, with bounded concurrency per downstream, timeouts on every call and a budget for the whole request

Why it works: Right, and the bound per downstream is what stops you from becoming their outage.

• Async suits many concurrent waits.
• Bound concurrency per dependency.
• Timeout every call, budget the request.
• Measure with real downstream latency.

You are asked to make a payments service active in two regions. What is the first design question?

The strongest answer: Which data must be strongly consistent and which can be eventually consistent — balances need one writer or a consensus store, receipts can replicate later; that split decides the architecture and most of the cost

Why it works: Right. Sort the data by consistency need first; everything follows from that.

• Classify data by consistency requirement.
• One writer or consensus for the strongly consistent part.
• Eventual replication for the rest.
• Cost follows the split.

Design multi-tenancy for a product where some tenants are regulated.

The strongest answer: Tiered isolation: shared schema with a tenant key and row-level enforcement for most, and a separate database or cluster for tenants whose contract or regulation demands it — one code path, an isolation level chosen per tenant, with noisy-neighbour limits and per-tenant keys for encryption

Why it works: Right. One product, several isolation tiers.

• Isolation as a per-tenant tier.
• Shared with row-level enforcement.
• Dedicated where required.
• Same code path.

Two-year platform strategy: what are the first three pages?

The strongest answer: The paved road (CI, identity, observe, deploy), the reliability bar (SLOs, incident), and the cost/IAM boundaries — not a tool shopping list

Why it works: Exactly. Road, bar, boundaries. Tools follow. Rewrites and vendor tables are appendices.

• Describe the paved road.
• Write the reliability bar.
• Write cost and IAM boundaries.
• Put vendors in the back.

Your worker cannot keep up with the producer and memory climbs. What is the right answer?

The strongest answer: Backpressure — a bounded queue that makes the producer wait or shed load, so the system degrades predictably instead of consuming memory until it dies

Why it works: Exactly. A bounded queue turns an outage into a slowdown you can see.

• Bounded queues create backpressure.
• Unbounded queues move the failure to memory.
• Decide explicitly: wait, shed or drop.
• Make the decision observable.

What is a cell-based architecture and when is it worth the complexity?

The strongest answer: Running many small, independent copies of the whole stack, each serving a slice of customers, so a failure or a bad deploy hurts one cell rather than everyone; worth it once a single shared stack's blast radius is the biggest risk you have

Why it works: Right. Blast radius is the product; complexity is the price.

• Many independent full-stack copies.
• Customers routed to a cell.
• Failure contained to one cell.
• Adopt when blast radius is the top risk.

How do you limit the blast radius of a failure in a system serving millions?

The strongest answer: Cells: independent copies of the whole stack, each serving a slice of customers, with a thin routing layer; a bad deploy or a poison request takes out one cell, deploys roll cell by cell, and capacity grows by adding cells — at the cost of cross-cell features being hard

Why it works: Right. Isolation by cell; the cost is anything cross-cell.

• Independent full-stack cells.
• Customers pinned to a cell.
• Deploy cell by cell.
• Cross-cell features are the cost.

What must a senior DevOps / platform engineer show besides a tool list?

The strongest answer: A reliability call, a blast-radius call, and a time they said no to a slogan (multi-cloud, five nines, “just SSH”) — tools move

Why it works: Exactly. Judgement on radius and slogans. A clean incident record is luck or silence.

• Hear a no to a slogan.
• Hear a blast-radius story.
• Hear a reliability trade.
• Do not hire on a tool list.

A Python service does heavy numeric work per request and latency is unacceptable. What options do you weigh?

The strongest answer: Push the loop into a library that releases the GIL, move the work to a process pool or a separate service, precompute or cache, or accept that this part does not belong in Python

Why it works: Right — and knowing when to leave Python for one component is part of the answer.

• Vectorised or native code releases the GIL.
• Process pools give real parallelism.
• Precompute or cache where the input repeats.
• Some components should leave Python.

Your DR runbook is a document. How do you make it real?

The strongest answer: Run it: scheduled game days where the team fails over to the secondary using only the runbook, timed against the RTO, with every gap fixed and the runbook updated afterwards; automate the steps that keep failing

Why it works: Right. A runbook that has not been run is a story.

• Execute the runbook on a schedule.
• Time it against RTO.
• Fix and update after every run.
• Automate the flaky steps.