All jobs / DevOps / SRE / cloud
Observability interview questions
Observability questions a DevOps / SRE / cloud can be asked, from first job to senior. Open one to see the strongest answer and why it works.
The server pings. Users still cannot log in. What was missing?
The strongest answer: A check that exercises the login path — or a metric on login errors — ping only says the box has a pulse
Why it works: Exactly. Synthetic or a metric on the user path. ICMP is not login.
- Monitor the user path.
- Ping is a pulse, not the product.
- Errors and latency on login.
- Do not stop at ICMP.
When do you use a metric instead of a log line?
The strongest answer: When I need a number over time I can alert on — rate, latency, errors — logs are for the story of one request
Why it works: Exactly. Metrics for the pulse, logs for the story. Grepping numbers is how you miss the page.
- Metrics: pulse and alerts.
- Logs: one request’s story.
- Do not grep a rate.
- You usually want both.
Describe the parts of an ELK or OpenSearch logging stack and what each does.
The strongest answer: Shippers on each host or pod (Filebeat, Fluent Bit) collect; Logstash or an ingest pipeline parses and enriches; Elasticsearch or OpenSearch indexes; Kibana or OpenSearch Dashboards searches and visualises; index lifecycle handles retention
Why it works: Exactly — collect, parse, index, query, retain. Naming the stages is how you locate a gap when logs go missing.
- Ship, parse, index, query.
- Agents collect; pipelines parse.
- The index is the store.
- Lifecycle handles retention.
What do you mean by observability in this interview?
The strongest answer: Enough metrics, logs, and traces to ask a new question without a new deploy — not a pile of dashboards we never open
Why it works: Exactly. Three signals, new questions. Dashboard sprawl is not observability.
- Metrics, logs, traces.
- Ask new questions without shipping.
- Do not confuse sprawl with signal.
- Do not leave DEBUG on.
On-call cannot find the one failing request in a sea of lines. What do you standardise?
The strongest answer: A correlation / request id, structured fields, and a default level — I do not invent a new log format per service
Why it works: Exactly. One id across the hop. Structure. No souvenirs of secrets.
- Correlation id.
- Structured fields.
- One time format.
- No secrets in logs.
You page on CPU > 80%. Users are fine. What do you change?
The strongest answer: Page on the user symptom (errors, latency, the SLO) — CPU is a cause you might graph, not a page unless you know it is the path
Why it works: Exactly. Symptoms over causes. A safe user and a loud pager is a bad SLO.
- Page the SLO / the user path.
- Graph causes.
- Do not page raw CPU by default.
- Do not mute everything.
How does Prometheus collect metrics, and why does a thirty-second batch job show none?
The strongest answer: Prometheus pulls: it scrapes each target’s /metrics endpoint on an interval, using service discovery; a job that exits in thirty seconds may never be scraped, so it pushes to the Pushgateway or reports through a long-lived collector
Why it works: Right — the pull model is the design choice everything else follows from, including how short-lived work is handled.
- Prometheus scrapes /metrics.
- Service discovery finds targets.
- Short jobs push to a gateway.
- Or go through a collector.
Why instrument services with OpenTelemetry instead of the observability vendor’s own SDK?
The strongest answer: A vendor-neutral API, SDK and collector: instrument once, export over OTLP to any backend, one context propagation across traces, metrics and logs, and switch vendors without touching application code
Why it works: Exactly — instrumentation is the expensive, sticky part; OpenTelemetry moves the vendor decision to the collector configuration.
- Instrument once, export anywhere.
- One API for traces, metrics, logs.
- Collector decides the backend.
- Propagation is standard.
Logs are 40% of the observability bill. What do you cut without going blind?
The strongest answer: High-volume low-value lines (health, debug, chatty access), keep the error path and the id — I do not drop all logs to save a slide
Why it works: Exactly. Cut chatty, keep the story of a failure. DEBUG-to-decide is a bigger bill.
- Drop health/debug volume.
- Keep errors and ids.
- Do not go log-blind.
- Do not DEBUG to decide.
You can afford traces on some paths. Which do you instrument first?
The strongest answer: The money and login paths, with a sample you can still debug — I do not trace every health check at 100% and then drop checkout
Why it works: Exactly. Instrument what you page on. 100% of everything is how you drop the useful 1%.
- Trace the SLO paths first.
- Sample the rest.
- Do not 100% health checks.
- Do not wait on a catalogue.
After a deploy, the Grafana dashboards time out and Prometheus runs out of memory. A developer added `user_id` as a label on the request counter.
What happened, and what is the rule?
The strongest answer: Every distinct label value is its own time series, so a user id made one series per user; labels are for bounded dimensions — route, status, region — and ids belong in logs and traces, linked by exemplars
Why it works: Right — cardinality is the cost model of a time-series database. Bounded labels, unbounded ids elsewhere.
- Each label value is a series.
- Labels: bounded dimensions only.
- Ids go to logs and traces.
- Watch series count after deploys.
What must a new service get on day one from the platform?
The strongest answer: An SLO template, a request id, a golden dashboard, and a page that hits a human — I do not let a service go live with SSH as the only tool
Why it works: Exactly. Paved observe. SSH-only is a night-watchman’s lantern.
- Id, dashboard, page.
- An SLO starter.
- No SSH-only prod.
- Do not unique-vendor each squad.