Over two years our team wrote eleven postmortems tagged "deploy". When we lined them up, almost none were caused by a bad feature. They came from the mechanics of shipping: a new instance taking traffic before it was ready, an old one killed mid-request, a column renamed while old code still read it, a CDN serving JavaScript that called an endpoint that no longer existed, and rollbacks that took 25 minutes because nobody had practiced them.

This is the checklist we built from those incidents. It applies whether you deploy to Kubernetes, ECS, a fleet of VMs behind a load balancer, or a PaaS. For each item we explain what goes wrong without it and what "done" looks like. If you can tick every box, a release becomes a non-event.

1. New instances only receive traffic when ready

A process that has started is not a process that can serve. It may still be loading configuration, warming caches, running JIT, or opening its database pool.

  • Expose a readiness endpoint that returns success only after startup work is complete and critical dependencies are reachable.
  • Point the load balancer or orchestrator at it, not at "is the port open".
  • Keep liveness separate and shallow. A liveness check that calls the database will restart every instance at once when the database blips, which turns a small incident into a big one.
# Kubernetes container spec (excerpt)
readinessProbe:
  httpGet: { path: /healthz/ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet: { path: /healthz/live, port: 8080 }
  periodSeconds: 10
  failureThreshold: 6
startupProbe:
  httpGet: { path: /healthz/live, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30   # up to 150 s to start before liveness takes over

Done when: you can start a new instance with an artificial 30-second startup delay and see zero errors at the load balancer.

2. Old instances finish what they started

Every rolling deploy stops old instances while they are serving traffic. Two things must happen in order: stop sending them new requests, then let them finish in-flight requests.

  • Handle SIGTERM: stop accepting new connections, finish in-flight requests, close keep-alive connections, then exit. Most frameworks have a graceful shutdown hook; make sure it is wired.
  • Give the platform time to remove the instance from routing before it stops listening. On Kubernetes that means a short preStop delay, because endpoint removal and SIGTERM happen concurrently.
  • Set the grace period longer than your slowest legitimate request plus the delay.
  • Configure the load balancer's deregistration or draining delay to match.

Background workers need the same care: stop pulling new jobs on SIGTERM, finish or re-queue the current one, and make jobs idempotent so a retry after a hard kill is safe.

Done when: a load test running during a deploy shows no 502, 503 or connection reset errors.

3. Database changes never break the running version

During a rolling deploy, old and new code run at the same time against the same database. Every schema change has to be compatible with both. The pattern that makes this routine is expand and contract:

Step Release Database Code
1 Expand Add new column, nullable or with a default Old code ignores it
2 Migrate Backfill in batches New code writes both old and new, reads old
3 Switch No change Code reads new, still writes both
4 Contract Drop old column Code no longer references it

A column rename done in one migration breaks whichever version is not expecting it. The four-step version takes four deploys, but each one is boring.

Other migration rules we enforce:

  • No long-running table rewrites in the deploy path. Large index builds use the online variant (CREATE INDEX CONCURRENTLY in PostgreSQL, online DDL or gh-ost in MySQL).
  • Every DDL session sets a short lock timeout so a blocked migration fails fast instead of queueing all traffic behind it.
-- PostgreSQL: fail after 5 s instead of blocking every query on the table
SET lock_timeout = '5s';
ALTER TABLE orders ADD COLUMN fulfilment_region text;
  • Migrations run as a separate step before the new code rolls out, not inside application startup where ten replicas race to run them.
  • Destructive steps (drops, constraint tightening) ship in a later release than the code change that made them safe.

Done when: you can deploy the new version, roll back to the old one, and the old one still works against the migrated schema.

4. Behaviour changes are decoupled from deploys

A deploy should change code; a flag should change behaviour. Feature flags let you ship code dark, turn it on for internal users, then a percentage, and turn it off in seconds without a deploy.

  • Wrap risky changes, new integrations and anything touching payments in a flag.
  • Give each flag an owner and a removal date. Stale flags become their own source of bugs.
  • Make the flag's "off" path the old, tested path, not an untested third branch.

Done when: the riskiest change in this release can be disabled without shipping code.

5. Clients and caches survive the transition

Users do not reload the page when you deploy. Their browser keeps old JavaScript, and your CDN may keep old HTML.

  • Use content-hashed asset filenames and keep the previous release's assets available for at least a day. Deleting them on deploy causes ChunkLoadError style failures for open tabs.
  • Keep API changes backward compatible for at least one release, or version them. An old SPA calling a renamed endpoint is a classic post-deploy error spike.
  • Version or namespace cache keys when the shape of cached data changes, so the new code does not deserialize the old format, and the old code does not choke on the new one during the rollout.
  • Mobile apps are the extreme case: old versions live for months, so API compatibility windows are long.

Done when: a browser tab opened before the deploy keeps working after it.

6. Rollback is one command and you have tried it

The time to discover that rollback is broken is not during an incident.

  • Keep previous artifacts (images, bundles) immutable and tagged. Never rebuild for a rollback.
  • Make rollback a single, documented action: re-deploy the previous image tag, kubectl rollout undo, or the equivalent in your platform.
  • Because of section 3, the previous version still works with the current schema, so rolling back code does not require rolling back the database.
  • Practice it. We roll back one ordinary release per month on purpose.
kubectl rollout history deployment/checkout-api
kubectl rollout undo deployment/checkout-api --to-revision=41
kubectl rollout status deployment/checkout-api --timeout=5m

Done when: anyone on call can roll back within five minutes without asking for help.

7. You know within minutes if the release is bad

  • Roll out progressively: one instance or a small percentage first, then the rest. Canary or blue-green if your platform supports it.
  • Watch a short list of signals during and after rollout: error rate, p95 latency, saturation, and one business metric such as checkouts per minute.
  • Annotate dashboards with deploy markers so a change in a graph lines up with a release.
  • Automate the stop: pause or abort the rollout when error rate crosses a threshold.

Done when: a deliberately broken canary is detected and halted without a human noticing first.

The checklist

Area Check
Readiness Traffic only after a readiness check passes; liveness is shallow
Draining SIGTERM handled; delay before shutdown; grace period longer than slowest request
Workers Stop consuming on shutdown; jobs idempotent
Migrations Expand and contract; online DDL; lock timeouts; run before rollout
Compatibility Old code works on new schema; APIs backward compatible for one release
Assets and caches Hashed assets kept for a day; cache keys versioned
Flags Risky behaviour behind a flag with an owner and expiry
Rollback Previous artifact immutable; one-command rollback; practiced monthly
Detection Progressive rollout; deploy markers; automatic abort on error spike

Common mistakes

Mistake Why it hurts Do this instead
Readiness probe that only checks the port Traffic hits instances still warming up Check real startup completion
Liveness probe that checks the database Mass restarts during a database blip Keep liveness process-local
Renaming a column in one migration Old or new code breaks during rollout Expand, migrate, switch, contract
Running migrations on app startup Replicas race; slow migration blocks startup Separate migration job before rollout
Deleting old assets on deploy Open tabs fail to load chunks Keep previous assets for a day or more
Rollback by rebuilding from an old commit Slow and not the same artifact Redeploy the previous immutable image

Key takeaways

  • Most deploy outages come from mechanics, not features; a checklist catches them.
  • Old and new versions always overlap, so every change must work with both.
  • Graceful shutdown needs coordination with routing, not just a signal handler.
  • Separate deploying code from enabling behaviour with feature flags.
  • A rollback you have not practiced is a hope, not a plan.

References