Continuous Delivery, GitOps & releases

Traditional CD pushes: your pipeline holds cluster credentials and runs kubectl apply. GitOps inverts it — an agent inside the cluster continuously pulls the desired state from Git and reconciles reality to match.

The four principles (OpenGitOps):

1. The system is declarative — everything is described as data, not scripts.
2. Desired state is versioned in Git — one auditable source of truth.
3. Changes are pulled automatically by an agent.
4. The agent continuously reconciles — drift is detected and corrected.

Why this is better:

• Git is the source of truth. To know what's in production, read the repo.
• Rollback is git revert. No special tooling, no panic.
• No cluster credentials in CI. The agent pulls, so your pipeline never holds production keys — a huge security win.
• Drift self-heals. If someone hand-edits the cluster at 2 a.m., it's reverted automatically.

Argo CD is the leading tool (Flux is the other). You point it at a repo of manifests/Helm charts, and its UI shows every app as Synced/OutOfSync and Healthy/Degraded.

The standard layout is two repos: application source code in one, deployment manifests in another. CI builds and pushes an image, then bumps the image tag in the config repo — and Argo CD deploys it.

Getting new code into production without breaking users is its own engineering discipline. The strategy you choose trades cost against risk.

• Recreate — stop the old, start the new. Simple, but causes downtime. Rarely acceptable.
• Rolling update — replace instances gradually (Kubernetes' default). No downtime, but during the rollout both versions serve traffic — so your changes must be backwards-compatible.
• Blue/green — run two complete environments. Blue is live; deploy to green, test it privately, then flip the load balancer. Rollback is flipping back — instant. Costs double the infrastructure during the switch.
• Canary — release to a small slice of traffic (1% → 5% → 25% → 100%), watching error rates and latency at each step. Automatically roll back if metrics degrade. The safest option for high-traffic systems; tools like Argo Rollouts and Flagger automate the ramp.

Underpinning all of them: feature flags. Deploy the code dark (shipped but switched off), then enable it for internal users, then 1% of customers. This decouples deploy from release — the single biggest reliability upgrade a team can make, because a bad feature is switched off in seconds without a redeploy.

And every strategy needs backwards-compatible database migrations (expand → migrate → contract), or rollback becomes impossible.

Code should not leap from a laptop to production. It walks through environments, each one a rehearsal that's closer to the real thing.

The usual ladder:

• Development — fast, disposable, broken often. Where you build.
• Staging / pre-production — as close to production as you can afford (same infra, same config shape, ideally anonymised production-like data). The last honest rehearsal.
• Production — real users, real money.

The rule that makes this meaningful: promote the same artifact. The exact image tested in staging is the one deployed to production — you never rebuild. What changes between environments is only configuration (via env vars, ConfigMaps, or Helm values files), never the code.

build once → app:1.5.0
   ↓ deploy to dev      (values-dev.yaml)
   ↓ deploy to staging  (values-staging.yaml)
   ↓ approval gate 🔒
   ↓ deploy to prod     (values-prod.yaml)

Guard production with an approval gate and keep secrets per-environment and separately scoped — a staging credential must never open production.

Increasingly teams also use ephemeral environments: a temporary environment spun up per pull request, torn down on merge. Reviewers get a live URL to click, which catches what a diff never will.