A checklist for zero-downtime deploys on Kubernetes

Setting the Deployment strategy to RollingUpdate does not mean no requests will be dropped during a rollout. Our team was seeing a few hundred 502s on every deploy, and the cause was three separate missing settings.

1. Readiness probe

If no readiness probeReadiness probe A Kubernetes health check that decides whether a pod is ready to receive traffic. is defined, a pod starts receiving traffic the moment it reaches Running. If the application has not yet set up its database connection pool, the first requests fail.

readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

2. Graceful shutdown

When a pod is deleted, Kubernetes kicks off two things at once: it sends SIGTERM and removes the pod from the endpoint list. The endpoint update can take a few seconds to reach every node. Requests arriving in that window land on a pod that is shutting down.

The sequence looks like this:

The simplest fix is a short preStop wait:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "10"]
terminationGracePeriodSeconds: 40

The application also needs to stop accepting new connections and finish in-flight requests when it receives SIGTERM. In Go, this is done with http.Server.Shutdown.

3. PodDisruptionBudget

During node maintenance, several pods can be evicted at the same time. A PDB limits this:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

Summary

After these three changes, the error count during deploys dropped to zero. We added the same settings as defaults to our new service template.