Observability & monitoring

You can't operate what you can't see. Observability rests on three pillars — metrics, logs, and traces — and metrics are where you start.

A metric is a number over time: request rate, error rate, latency, CPU, memory. The de-facto open-source stack is Prometheus (collects and stores metrics) plus Grafana (visualises them).

Prometheus works by scraping — it periodically pulls a /metrics endpoint that your apps expose. You then query with PromQL:

# 95th-percentile request latency over 5 minutes
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Grafana turns those queries into dashboards and, crucially, alerts — fire a notification when error rate crosses a threshold or latency spikes. A good starting framework is the RED method for services (Rate, Errors, Duration) and USE for resources (Utilization, Saturation, Errors).

The discipline: alert on symptoms users feel (high latency, errors), not on every low-level cause, or you'll drown in noise and start ignoring pages.

Metrics tell you that something is wrong; logs and traces tell you what and where.

Logs are timestamped event records. On one server you tail a file; across a fleet you ship them to a central system — the ELK stack (Elasticsearch, Logstash, Kibana) or Grafana Loki — so you can search everything in one place. Emit structured logs (JSON with fields) rather than free text, so they're queryable, and include a request ID to tie events together.

Traces follow a single request as it hops across many services — essential in microservices, where one user click might touch ten backends. A trace shows the whole journey and how long each hop took, so you can pinpoint which service caused the slowdown. Jaeger and Tempo visualise them.

OpenTelemetry (OTel) is the unifying standard: one vendor-neutral set of libraries and agents to emit metrics, logs, and traces. Instrument with OTel once and you can send data to any backend, avoiding lock-in.

"Reliable" is meaningless until you make it a number. That's what SLIs, SLOs, and error budgets do — they turn reliability into an engineering decision instead of an argument.

• An SLI is a measured indicator of health: e.g. the proportion of requests that are both successful and faster than 300 ms.
• An SLO is your target for that SLI over a window: e.g. 99.9% over 30 days. Note that 99.9% still allows ~43 minutes of "bad" time a month — 100% is the wrong target; it's infinitely expensive.
• The error budget is what's left: 100% − SLO. That budget is a currency you get to spend.

The mechanism is elegant: if you have budget remaining, you can take risks and ship features fast. If you've burned it, you stop shipping and invest in reliability until you're back in the black. It aligns product and ops around one shared number, and it tells you when to page a human — when you're burning budget fast enough to blow the SLO.

Monitoring that pages you for everything is worse than no monitoring — because you learn to ignore it. Alert fatigue is the number-one cause of missed real incidents.

Alert on symptoms, not causes. A page should mean users are being hurt right now: error rate is up, latency is up, the queue is backing up beyond recovery. "CPU is at 90%" is not an alert — if CPU is high and users are fine, nothing is wrong. Put causes on dashboards; put symptoms on pagers.

Every page must be actionable and urgent. The test: if the human can't do anything about it, or it can wait until Monday, it is not a page — make it a ticket or a dashboard. A good alert links to a runbook: what this alert means, how to confirm it, what to check, how to mitigate.

Tie alerts to SLOs: rather than static thresholds, alert on error-budget burn rate — "we're consuming the month's budget in hours." That fires for the things that genuinely threaten your objective, and stays quiet for harmless blips.

On-call must be humane and sustainable: a fair rotation, a real handover, compensation, and — critically — the authority and time to fix the things that woke you. If the same alert fires every week and nobody's allowed to fix the cause, the rotation is broken.

Metrics say "checkout is slow." Traces say "it's the payment service." Profiling says "it's this function, on line 42, allocating memory in a loop." That last mile is where the fix lives.

APM (Application Performance Monitoring) — Datadog, New Relic, Grafana Pyroscope, Elastic APM — instruments your app and shows request throughput, error rates, slow transactions, and a breakdown of where the time actually went (database, external call, CPU).

Learn to read a flame graph: the x-axis is time spent (wider = more time), and the y-axis is the call stack. The widest bar is your bottleneck. It's the fastest way to find a performance problem in an unfamiliar codebase.

The usual culprits, in the order you should suspect them:

1. The N+1 query — code that runs one database query per row in a loop. It's fast with 10 rows in dev and catastrophic with 10,000 in production. This is the most common performance bug in the world.
2. A missing database index — a full table scan on a big table.
3. A missing cache on a hot, expensive path.
4. Blocking I/O on the request thread.
5. Memory leaks and excessive garbage collection.

Continuous profiling runs in production at low overhead, so you can look at yesterday's incident rather than trying to reproduce it.