Production Operations

Rolling Cluster Upgrades

● Advanced ⏱ 15 min read

Kubernetes releases a new minor version every four months. Each minor version is supported for approximately 14 months. Falling behind means missing security patches and losing support. Staying current means upgrading production clusters reliably, repeatedly, and without user impact. This guide covers the rules, the order, and the runbook.

Version Skew Policy

Kubernetes enforces strict version compatibility rules between components. Violating them causes subtle failures that are hard to diagnose.

Component pair	Allowed skew
kube-apiserver ↔ kube-controller-manager / kube-scheduler	Within 1 minor version of apiserver (apiserver must be ≥)
kube-apiserver ↔ kubelet	kubelet can be up to 3 minor versions behind apiserver
kube-apiserver ↔ kube-proxy	kube-proxy can be up to 3 minor versions behind apiserver
kubectl ↔ kube-apiserver	kubectl can be 1 minor version ahead or behind
Control plane nodes (HA)	All control plane nodes must be within 1 minor version of each other during upgrade

⚠️

Upgrade one minor version at a time

Kubernetes does not support skipping minor versions during an upgrade. If you are on 1.28, you must upgrade to 1.29, then 1.30. You cannot jump from 1.28 to 1.30 directly.

Upgrade Order

Cluster upgrade sequence — always control plane first

Control plane node 1 (primary)

Upgrade kube-apiserver, controller-manager, scheduler, etcd

Control plane nodes 2, 3 (HA)

Upgrade one at a time; wait for Ready between each

Worker nodes — rolling

Drain → upgrade kubelet + kube-proxy → uncordon; one node at a time

Add-ons

CNI, CoreDNS, kube-proxy DaemonSet, cluster-autoscaler, metrics-server

Workloads are never touched during a cluster upgrade. Pods keep running on old nodes during the upgrade; they are only rescheduled when their node is drained.

Control plane first, workers rolling. Workers can be up to 3 minor versions behind the upgraded control plane during the node rollout window.

Managed Clusters

On EKS, GKE, and AKS the control plane upgrade is handled by the cloud provider — you initiate it and the provider handles etcd, apiserver, and the static pods. You still own the node group upgrade.

EKS — upgrade control plane then node group

# Upgrade control plane (takes ~10 min)
aws eks update-cluster-version \
  --name production \
  --kubernetes-version 1.31 \
  --region us-east-1

# Watch until ACTIVE
aws eks describe-cluster --name production \
  --query 'cluster.{status:status,version:version}'

# Upgrade managed node group (rolling replacement of EC2 instances)
aws eks update-nodegroup-version \
  --cluster-name production \
  --nodegroup-name workers \
  --kubernetes-version 1.31

GKE — upgrade control plane then node pools

# Upgrade control plane
gcloud container clusters upgrade production \
  --master \
  --cluster-version 1.31 \
  --zone us-central1-a

# Upgrade node pool (surge upgrade: +1 node, drain, repeat)
gcloud container clusters upgrade production \
  --node-pool default-pool \
  --cluster-version 1.31 \
  --zone us-central1-a

Self-Managed — kubeadm

kubeadm upgrade — control plane node

# 1. Upgrade kubeadm itself first
apt-get update && apt-get install -y kubeadm=1.31.0-00

# 2. Verify the upgrade plan
kubeadm upgrade plan

# 3. Apply the upgrade (upgrades apiserver, controller-manager, scheduler, etcd)
kubeadm upgrade apply v1.31.0

# 4. Upgrade kubelet and kubectl on the control plane node
apt-get install -y kubelet=1.31.0-00 kubectl=1.31.0-00
systemctl daemon-reload && systemctl restart kubelet

kubeadm upgrade — each worker node

# From a control-plane node: drain the worker
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

# On the worker node itself:
apt-get update && apt-get install -y kubeadm=1.31.0-00
kubeadm upgrade node

apt-get install -y kubelet=1.31.0-00
systemctl daemon-reload && systemctl restart kubelet

# Back on control plane: uncordon the worker
kubectl uncordon node-1

Drain & Cordon

kubectl cordon marks a node as unschedulable — no new pods land on it, but existing pods keep running. kubectl drain cordons the node and then evicts all non-DaemonSet pods, waiting for them to reschedule elsewhere before returning.

# Cordon only — stop new pods, existing pods stay
kubectl cordon node-1

# Drain — cordon + evict pods (respects PodDisruptionBudgets)
kubectl drain node-1 \
  --ignore-daemonsets \          # DaemonSet pods cannot be rescheduled
  --delete-emptydir-data \       # pods using emptyDir lose the data
  --grace-period=60 \            # give pods 60s to terminate gracefully
  --timeout=300s                 # fail if drain takes more than 5 min

# Uncordon after node is upgraded and rejoins
kubectl uncordon node-1

💡

Drain respects PodDisruptionBudgets

If a Deployment has a PDB that requires at least 2 replicas always available, drain will wait rather than evict the pod that would violate it. This is the safety mechanism — PDBs prevent a drain from taking a service below minimum capacity. If drain hangs, check your PDBs.

Pre-Upgrade Checks

# Check current cluster version
kubectl version --short

# Check all node versions (kubelet)
kubectl get nodes -o wide

# Find deprecated API usage (resources using APIs removed in next version)
# Install pluto: https://github.com/FairwindsOps/pluto
pluto detect-all-in-cluster

# Check add-on compatibility (cert-manager, ingress-nginx, etc.)
# Review each add-on's compatibility matrix before upgrading

# Verify all nodes are Ready
kubectl get nodes | grep -v Ready

# Check for any pods in a bad state
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded | grep -v Completed

# Take an etcd snapshot before upgrading (self-managed only)
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-pre-upgrade.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key

Rollback Planning

Control plane rollback is not supported by kubeadm once applied. Rollback options are:

etcd snapshot restore — restore the etcd database to the pre-upgrade snapshot. Only works for self-managed clusters. Restores all cluster state to the snapshot point.
Worker node rollback — nodes that have not yet been drained can be kept on the old version (up to 3 minor versions behind). Roll back the node pool upgrade before draining those nodes.
Managed cluster rollback — EKS/GKE/AKS control plane rollback is generally not supported. Create a new cluster and migrate workloads if the upgrade fails.

The practical rollback plan for most teams: test upgrades in staging first, take an etcd snapshot before upgrading, and have a workload migration runbook ready. The cluster is easier to recreate than to roll back.

kubectl Commands

# Cluster version
kubectl version --short

# All node versions
kubectl get nodes -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion

# Check API server version
kubectl get --raw /version | jq .

# Verify add-on pod versions after upgrade
kubectl get pods -n kube-system -o wide | grep -E "coredns|kube-proxy|metrics-server"

# Watch node status during rolling upgrade
watch kubectl get nodes