Kubernetes & orchestration

Kubernetes (K8s) runs containers across a fleet of machines and keeps them healthy without you babysitting them. It's declarative: you describe the desired state, and Kubernetes continuously works to make reality match — the reconciliation loop at the heart of everything.

A cluster has two halves:

• Control plane — the brain: the API server (front door), etcd (the database of desired state), the scheduler (places workloads), and controllers (drive reality toward desired state).
• Nodes — the workers that actually run your containers.

The core objects, from small to large:

• Pod — one or more containers that share a network/storage; the smallest deployable unit.
• ReplicaSet — keeps N copies of a Pod running.
• Deployment — manages ReplicaSets and gives you rolling updates and rollbacks.
• Service — a stable network address and load balancer for a set of Pods (Pods are ephemeral; Services are not).

You interact via kubectl: kubectl get pods, kubectl describe, kubectl apply -f app.yaml, kubectl logs.

Deployment is not the only way to run something. Kubernetes has a workload type for each shape of job, and picking the wrong one is a classic beginner mistake.

• Deployment — for stateless apps (web servers, APIs). Pods are interchangeable; any one can be killed and replaced. Gives rolling updates and rollbacks. This is your default.
• StatefulSet — for stateful apps (databases, Kafka, Elasticsearch). Pods get a stable identity (db-0, db-1) that survives restarts, stable storage attached to each, and ordered startup/shutdown. A database in a Deployment will corrupt itself; use a StatefulSet.
• DaemonSet — runs exactly one Pod on every node. Perfect for node-level agents: log collectors (Fluent Bit), monitoring (node-exporter), network plugins. Add a node, it gets the Pod automatically.
• Job — runs a Pod to completion and stops. For migrations, batch processing, one-off tasks. It retries on failure.
• CronJob — a Job on a schedule (same cron syntax you already know). For nightly backups and reports.

Supporting these: init containers run to completion before your app container starts (wait for a database, run a migration), and liveness/readiness probes tell Kubernetes whether a Pod is alive and whether it's ready for traffic.

Once Pods run, three practical concerns decide whether your app actually works in Kubernetes.

Configuration: never bake config into images. Inject it with ConfigMaps (plain settings) and Secrets (passwords, tokens — base64-encoded, and ideally encrypted at rest). Your Pods read them as environment variables or mounted files, so the same image runs in dev and prod with different config.

Networking: every Pod gets an IP, but Pods die and get replaced, so you never talk to them directly. A Service gives a stable virtual IP and DNS name that load-balances across healthy Pods. To expose HTTP to the outside world, an Ingress (or the newer Gateway API) routes external traffic by hostname/path to the right Service — one entry point, TLS termination included.

Storage: containers are ephemeral, so for databases you need PersistentVolumes and PersistentVolumeClaims, which attach durable disks that survive Pod restarts.

Finally, namespaces partition a cluster for different teams/environments, and RBAC controls who can do what — essential once more than one person shares a cluster.

Kubernetes can scale for you — but only if you tell it what your Pods need. That starts with resource requests and limits:

resources:
  requests: { cpu: "100m", memory: "128Mi" }   # what the scheduler reserves
  limits:   { cpu: "500m", memory: "512Mi" }   # the hard ceiling

Requests decide where a Pod is scheduled (the node must have that much free). Limits cap usage — exceed the memory limit and your container is OOMKilled; exceed CPU and it's throttled. Set no requests and the scheduler is flying blind, which causes most "random" cluster instability.

Then three autoscalers, operating at different levels:

• HPA (Horizontal Pod Autoscaler) — adds/removes Pod replicas based on CPU, memory, or custom metrics (e.g. requests per second). This is the one you'll use most: handle more traffic by running more copies.
• VPA (Vertical Pod Autoscaler) — adjusts the requests/limits of a Pod, i.e. right-sizes it. Useful for tuning, and it conflicts with HPA on the same metric — don't run both on CPU.
• Cluster Autoscaler — adds/removes nodes when Pods can't be scheduled for lack of capacity. HPA is useless without it once the cluster is full.

Writing raw YAML for every environment gets repetitive fast — you end up copy-pasting near-identical manifests for dev, staging, and prod. Helm is the package manager for Kubernetes that fixes this.

A Helm chart is a templated, versioned bundle of manifests. You keep one template and supply a values.yaml per environment:

helm install myapp ./chart -f values-prod.yaml
helm upgrade myapp ./chart --set replicaCount=5
helm rollback myapp 1     # instant rollback to a previous release

Charts make deployments reusable, parameterised, and versioned — and there are public charts for common software (Postgres, Redis, Prometheus) so you don't reinvent them.

Managed Kubernetes is the other half of being productive. Running your own control plane is hard and thankless, so use a cloud offering — EKS (AWS), GKE (Google), or AKS (Azure). They operate the control plane, handle upgrades, and integrate with cloud load balancers and storage, leaving you to focus on your workloads.

Kubernetes fails in a small number of recognisable ways. Learn the loop and the top errors and you can debug almost anything.

The loop — always in this order:

kubectl get pods                 # what state is it in?
kubectl describe pod <name>      # WHY — read the Events at the bottom
kubectl logs <name>              # what did the app itself say?
kubectl logs <name> --previous   # logs from the crashed instance
kubectl exec -it <name> -- sh    # go inside and look around
kubectl get events --sort-by=.metadata.creationTimestamp

describe is the one beginners skip and experts read first — the Events section usually states the problem in plain English.

The errors you will actually see:

• ImagePullBackOff / ErrImagePull — Kubernetes can't fetch the image. Wrong name/tag, or missing registry credentials (imagePullSecrets).
• CrashLoopBackOff — the container starts, dies, and restarts in a loop. It's an app problem: read logs --previous. Usually a bad config, a missing env var, or an unreachable dependency.
• Pending — the scheduler can't place it: not enough CPU/memory on any node, or an unsatisfiable node selector / taint.
• OOMKilled — it exceeded its memory limit. Raise the limit or fix the leak.
• Service returns nothing — the Service's selector doesn't match the Pod's labels. Check with kubectl get endpoints <svc>: empty endpoints means the selector is wrong.

Kubernetes' real power isn't the built-in objects — it's that you can teach it new ones. This is how the ecosystem (Prometheus, cert-manager, database operators) extends the platform.

Two pieces:

• A CRD (Custom Resource Definition) adds a new object type to the Kubernetes API. Define a Database kind, and kubectl get databases starts working as if it were built in.
• An Operator is a controller that watches those objects and makes reality match them. It's the same reconciliation loop Kubernetes uses internally, but carrying your operational knowledge.

The insight: an operator encodes what a human expert would do. A Postgres operator doesn't just start a container — it provisions storage, configures replication, takes scheduled backups, handles failover, and performs safe version upgrades. You declare:

apiVersion: acid.zalan.do/v1
kind: postgresql
spec:
  numberOfInstances: 3
  version: "16"

…and the operator does everything a DBA would, continuously.

You'll use operators long before you write one (install them via Helm). When you do write one, frameworks like Kubebuilder or the Operator SDK generate the scaffolding in Go.