Reliability & SRE

Reliability engineering is what separates "it deployed" from "it stays up." Failure is inevitable, so you engineer to absorb and recover from it rather than pretending it won't happen.

Incident response is a discipline: detect fast (good alerting), have a clear on-call rotation, assign an incident commander for big ones, communicate status, and — most importantly — write a blameless postmortem afterward. Blameless means you fix the system and process, not punish the person; people are honest only when they're not afraid, and honesty is how you actually learn.

To find weaknesses before users do, practise chaos engineering: deliberately inject failure (kill a server, add latency, cut a dependency) in a controlled way and verify the system copes. Netflix's Chaos Monkey made this famous. It sounds reckless; done well it's the opposite — it turns "we think we're resilient" into evidence.

Design for failure with timeouts, retries with backoff, circuit breakers, and graceful degradation, so one broken dependency doesn't cascade into a full outage.

You do not know your system is resilient until you've broken it on purpose. Chaos engineering is the practice of injecting failure deliberately, in a controlled way, to find weaknesses before they find you at 3 a.m.

It is not "randomly break production." It's a scientific method:

1. Define steady state — a measurable signal of healthy (e.g. checkout success rate > 99%).
2. Hypothesise: "If one payment-service Pod is killed, the success rate will not change."
3. Inject the smallest failure — kill one Pod. Start in staging; limit the blast radius in production to a tiny slice of traffic.
4. Measure. Did steady state hold?
5. Learn and fix. If it broke, you've found a real bug cheaply, on your terms, during working hours.

Netflix's Chaos Monkey (which kills random instances continuously) made the idea famous; Litmus, Chaos Mesh, and AWS FIS are modern tools. Typical experiments: kill instances, add network latency, drop a dependency, fill a disk, exhaust CPU, simulate an entire AZ failure.

The findings are always the same shape and always valuable: a missing timeout, a retry storm, a hidden single point of failure, a health check that lies, an alert that never fired.

A gentler on-ramp is a game day — a scheduled, human-run exercise where the team simulates an outage and practises the response.

Everything above is worthless if you lose the data. Backups are the last line of defence — and an untested backup is not a backup.

Two numbers define your strategy, and the business must set them:

• RPO (Recovery Point Objective) — how much data can you afford to lose? Hourly backups mean an RPO of one hour.
• RTO (Recovery Time Objective) — how long can you afford to be down? This drives how much standby infrastructure you pay for.

The classic rule is 3-2-1: three copies, on two different media, one off-site — and, in the ransomware era, one immutable/offline copy that a compromised admin account cannot delete. Automated backups that your production credentials can erase will be erased by an attacker.

Test the restore. On a schedule. The graveyard of companies is full of teams whose backups had been silently failing for months, or who discovered the restore takes 40 hours. A backup you have never restored is a hope, not a plan.

Disaster recovery scales the same idea to a whole region, and the tiers cost accordingly:

• Backup & restore — cheapest, slowest (hours–days).
• Pilot light — core systems (like a replicated database) always running, the rest started on demand.
• Warm standby — a scaled-down full copy, ready to scale up (minutes).
• Multi-site active/active — full capacity in two regions, near-zero RTO, roughly double the cost.

Write the runbook and rehearse the failover — during an actual disaster is the worst time to read your own documentation.

"Will it survive launch day?" is a question you should be able to answer with data, not vibes.

Load testing is how you get that data — and there are distinct kinds, each answering a different question:

• Load test — expected peak traffic. Does it hold up?
• Stress test — push past the limit. Where does it break, and does it fail gracefully (shedding load, returning 503s) or catastrophically (falling over and corrupting things)?
• Soak test — sustained load for hours. This is how you catch memory leaks and connection-pool exhaustion, which look fine for ten minutes and die overnight.
• Spike test — instant 10× traffic. Can your autoscaler react in time? (Usually not as fast as you hope — scaling takes minutes; a spike takes seconds. This is why you pre-scale for known events.)

Tools: k6, Gatling, Locust, JMeter. Run them in CI against staging so a performance regression is caught in a PR, not by users.

Capacity planning turns the results into decisions: measure your headroom (how much spare capacity you run with), find the bottleneck (CPU, memory, database connections, IOPS, a third-party API's rate limit), and model growth. A well-run service typically targets ~50% utilisation at peak so it can absorb a surge and lose a node without drama.

Then remember: performance is a feature, and it is also a cost. The cheapest capacity is code that does less work — an index, a cache, a fixed N+1 query is often worth more than doubling the servers.