Kubernetes

The 30-second version

That second half is the whole idea. Kubernetes (often written k8s: k, then 8 letters, then s) is an OrchestratorOrchestratorA system (like Kubernetes) that starts, stops, restarts and moves your app's instances for you, and tells the load balancer which ones exist.. It notices when reality drifts from what you asked for, because a container crashed or a machine died, and fixes it without waking anyone up.

Try being the bad luck. Each box below is one machine (Kubernetes calls a machine a node), and each green chip is one running copy of the app. The controller checks every second. Kill pods, take a whole machine down, change the number you asked for, and watch it bring the kitchen back to "3 running":

What you'll learn

1. Why you'd want this at all, and what it replaces (§02).
2. The parts of a cluster: which program does what when you type kubectl apply (§03).
3. The four objects you'll use every day: Pods (§04), Deployments (§05), Services (§06), and Ingress for outside traffic (§07).
4. Making apps behave well inside it: config and secrets (§08), health probes (§09), CPU and memory budgets (§10).
5. Shipping: rolling updates and rollbacks (§11), autoscaling and namespaces (§12).
6. When it breaks: the debugging workflow (§13), then production lessons (§14).

Why an orchestrator?

Start with one server

Say you've built a small Go web service and packaged it as an image, hello:v1. On one server, running it is one command:

docker run -d --restart=always -p 80:8080 hello:v1

That's genuinely fine for a side project. Docker even restarts the container if it crashes (--restart=always). Now the service gets popular, and the list of questions grows.

Then reality arrives

Every row is a small program someone has to write, run and keep correct. Kubernetes is those programs, already written, plus one shared database of "what should be running", which the programs all read.

The vocabulary, in one breath

You'll see these words on every page from here on. Each gets its own section later; for now, just the shape:

• A clusterCluster (Kubernetes)A group of machines that Kubernetes manages as one big computer: a control plane that makes decisions, plus worker nodes that actually run your containers. is a group of machines managed together. Each machine is a nodeNode (Kubernetes)One machine in a Kubernetes cluster — a real server or, more often, a cloud virtual machine. Each node runs a kubelet and a container runtime, and hosts some of the cluster's pods..
• A podPod (Kubernetes)The unit Kubernetes runs and restarts: one or a few containers that live and die together. Stopping a pod is how Kubernetes stops your process during a deploy. is one running copy of your app: usually one container, plus its own IP address.
• A DeploymentDeployment (Kubernetes)A Kubernetes object that keeps a chosen number of identical pods running and replaces them gradually when you change the image or settings. says "keep N pods of this image running" and handles version upgrades.
• A ServiceService (Kubernetes)A stable name and virtual IP inside a Kubernetes cluster that load-balances traffic to whichever pods currently match its label selector. gives those pods one stable name and spreads requests across them.
• You describe all of this in YAMLYAMLA text format for configuration that uses indentation instead of brackets. Used by GitHub Actions, Kubernetes, Docker Compose and many other tools; indentation mistakes change the meaning. files and send them to the cluster with kubectlkubectlThe command-line tool for talking to a Kubernetes cluster (say “cube-control” or “cube-cuttle”). Every command — get, apply, logs, delete — is really an HTTP request to the cluster's API server..

When you don't need it

Kubernetes is powerful and also a lot of moving parts. One or two services with steady traffic often do better on a simpler platform: a PaaSIaaS / PaaS / SaaSHow much of the stack the cloud runs for you: IaaS rents raw machines and networks, PaaS runs your code on a managed platform, SaaS is a finished app you just use. (Heroku-style "push code, it runs"), a managed container service like AWS ECS or Google Cloud Run, or even one VM with Docker Compose. Those hide the cluster from you entirely.

Kubernetes earns its keep when you have many services, several teams, or traffic that changes a lot, and you want one consistent way to deploy, scale and heal all of it. Most companies don't run the cluster's brain themselves either: they rent it as managed KubernetesManaged Kubernetes (EKS, GKE, AKS)A cloud provider runs and upgrades the control plane for you (Amazon EKS, Google GKE, Azure AKS); you pay for and manage the worker nodes (or not, in their serverless modes). (EKS, GKE, AKS). Cloud trade-offs are covered in Cloud basics.

Anatomy of a cluster

The control plane

• The API serverAPI server (kube-apiserver)The front door of the cluster: the only component that reads and writes the cluster database (etcd). kubectl, the scheduler, controllers and kubelets all talk to it over HTTPS. is a big HTTPS APIAPIApplication Programming Interface: the set of functions, types or HTTP endpoints that one piece of software offers for others to call.. kubectl, your CI pipeline, the scheduler and every kubelet are all just clients of it. It checks who you are and whether you're allowed (RBACRBAC (role-based access control)Granting permissions to roles (admin, editor, viewer) and roles to users, instead of granting each permission to each user directly.), validates the object, and saves it.
• etcdetcdAn open-source, Raft-based key-value store used for configuration, leader election and locks. Kubernetes keeps all of its cluster state in etcd. is where it saves things: a small, very reliable key-value database. In production it runs as 3 or 5 copies that agree on every write using RaftRaftA popular consensus algorithm designed to be understandable: nodes elect a leader, the leader appends commands to a log, and an entry counts once a majority has stored it. Used by etcd, Consul, CockroachDB., so losing one machine loses no data.
• The schedulerScheduler (kube-scheduler)The control-plane component that picks a node for every new pod, based on how much CPU and memory the pod asks for and what each node has free. It only decides; the node's kubelet starts the pod. watches for pods that have no node yet and picks one, based on how much CPU and memory each pod asks for (§10) and what each node has free.
• The controller manager runs the controllersController (control loop)A program that runs forever in a loop: look at what you asked for (desired state), look at what exists (actual state), and act to close the gap. Kubernetes is built from dozens of them.: one loop per kind of object, each doing "compare wanted with actual, fix the gap".

Each node

• The kubeletkubeletThe agent that runs on every node. It watches the API server for pods assigned to its node, asks the container runtime to start their containers, runs their health probes and reports their status back. is the agent on every node. It watches for pods assigned to its node, asks the container runtime to start them, runs their health checks (§09) and reports back.
• The container runtimeContainer runtime (runc, containerd)The low-level software that asks the Linux kernel to create a container. runc makes the namespaces and cgroups and starts your process; containerd manages images and runc for Docker and Kubernetes. (usually containerd) does the actual work of pulling images and starting containers, the same machinery Docker uses underneath.
• kube-proxykube-proxyA small program on every node that turns Services into network rules (usually iptables or nftables rules in the Linux kernel), so traffic sent to a Service's IP is forwarded to one of its pods. programs the node's network so a Service's address reaches a real pod (§06).

Seeing it for real

kubectl is the command-line client. Here's our one-node kindkind (Kubernetes IN Docker)A tool that runs a whole Kubernetes cluster inside Docker containers on your laptop: each “node” is a container. Made for testing; not for production. (Not to be confused with the kind: field in YAML.) cluster. In kind, the "node" is a Docker container pretending to be a machine:

$ kubectl get nodes -o wide
NAME                      STATUS   ROLES           AGE   VERSION   INTERNAL-IP   OS-IMAGE                       CONTAINER-RUNTIME
atlas-k8s-control-plane   Ready    control-plane   31s   v1.37.0   172.18.0.2    Debian GNU/Linux 13 (trixie)   containerd://2.3.4
# (two columns trimmed to fit)

The control plane itself runs as pods, in a namespaceNamespace (Kubernetes)A named folder for objects inside one Kubernetes cluster (e.g. billing-prod). Permissions and Secrets are scoped to it: a pod can only use Secrets from its own namespace. (a named folder for objects) called kube-system:

$ kubectl get pods -n kube-system
NAME                                              READY   STATUS    RESTARTS   AGE
coredns-559f6c778d-rw95f                          1/1     Running   0          22s
coredns-559f6c778d-wzzfq                          1/1     Running   0          22s
etcd-atlas-k8s-control-plane                      1/1     Running   0          27s
kindnet-9tsch                                     1/1     Running   0          22s
kube-apiserver-atlas-k8s-control-plane            1/1     Running   0          27s
kube-controller-manager-atlas-k8s-control-plane   1/1     Running   0          29s
kube-proxy-z67bk                                  1/1     Running   0          22s
kube-scheduler-atlas-k8s-control-plane            1/1     Running   0          27s

Every box from the diagram is there, plus two more. coredns is the cluster's DNSDNSThe Domain Name System: the internet's phone book that turns a name like example.com into an IP address your computer can connect to. server (§06), and kindnet is the pod networkPod network (CNI)The network that gives every pod its own IP address and lets any pod reach any other pod directly, even on another node. It's set up by a network plugin following the CNI standard (Calico, Cilium, kindnet…). plugin, which gives every pod its own IP address. (Only the kubelet and containerd are missing from the list: they run directly on the node, not as pods.)

What happens when you kubectl apply

Here's the central trick of Kubernetes, step by step. No component tells another what to do. Each one watches the API server and reacts when something it cares about changes:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: hello
5spec:
6  replicas: 3
7  selector:
8    matchLabels:
9      app: hello
10  template:
11    metadata:
12      labels:
13        app: hello
14    spec:
15      containers:
16        - name: hello
17          image: hello:v1

Notice that the Deployment controller never talks to the scheduler, and the scheduler never talks to the kubelet. They only read and write objects through the API server. That's why any one of them can crash and restart without losing anything: the state lives in etcd, and the loop just picks up where it left off.

Pods: the smallest thing Kubernetes runs

The app we'll deploy

Throughout this topic we deploy one tiny web server. (The Go is basic; if Go syntax is new to you, the 5-minute primer in Async / Await §02 covers it.) It answers with a greeting, the name of the machine it's running on, and its version. That will let us see which copy answered each request. Here is a simplified version in Go and in Node. (The Go server we actually deployed also has health endpoints and a clean shutdown on SIGTERM, which §09 and §14 add.)

package main

import (
	"fmt"
	"net/http"
	"os"
)

func main() {
	pod, _ := os.Hostname() // ← inside Kubernetes, the hostname is the pod's name
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "hello from %s (v1)\n", pod)
	})
	http.ListenAndServe(":8080", nil) // listen on every interface, port 8080
}

We built the Go version into a 2.4 MB image, hello:v1, using a FROM scratch Dockerfile (an image with nothing in it but our binary; see Containers §07 for how). In kind you load a local image straight into the cluster's node with kind load docker-image hello:v1, so no registryContainer registryA server that stores container images by name and tag (Docker Hub, GitHub Container Registry, Amazon ECR), so CI can push an image once and every server can pull the same one. is needed. A real cluster pulls images from a registry instead.

Your first manifest

You describe what you want in a manifestManifestA YAML (or JSON) file describing one or more Kubernetes objects — their kind, name and spec. You hand it to kubectl apply; it's usually kept in git next to the code., a YAML file. Every Kubernetes objectObject (Kubernetes)One record stored in the cluster: a Pod, a Deployment, a Service, a ConfigMap… Each has a kind, a name, a spec (what you want) and a status (what Kubernetes last saw). has the same four top-level parts:

apiVersion: v1          # which version of the Kubernetes API defines this kind
kind: Pod               # what type of object this is
metadata:
  name: hello-pod       # its name (unique per kind, per namespace)
  labels:
    app: hello          # ← a tag other objects can search for (§05, §06)
spec:                   # what you want: for a Pod, its containers
  containers:
    - name: hello
      image: hello:v1
      ports:
        - containerPort: 8080   # documentation: the app listens here

Send it to the cluster with kubectl apply, then look:

$ kubectl apply -f pod.yaml
pod/hello-pod created
$ kubectl get pods -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP           NODE                      NOMINATED NODE   READINESS GATES
hello-pod   1/1     Running   0          5s    10.244.0.5   atlas-k8s-control-plane   <none>           <none>

READY 1/1 means one of one containers is running and ready (ready = passing its readiness check, which §09 covers). You can ignore the last two columns for now. The pod has its own IP address, 10.244.0.5, reachable from any other pod in the cluster. That's the pod networkPod network (CNI)The network that gives every pod its own IP address and lets any pod reach any other pod directly, even on another node. It's set up by a network plugin following the CNI standard (Calico, Cilium, kindnet…). at work.

Why "pod" and not just "container"?

A pod is a small wrapper that can hold more than one container, and the containers in one pod share things:

• One network. They share an IP address and can reach each other on localhost. (In Linux terms they share one network namespaceNamespaces (Linux)A kernel feature that gives a process its own private view of something — its own list of PIDs, its own network interfaces, its own file system root. It controls what a container can see..)
• Volumes. They can mount the same folders.
• One fate. They're scheduled onto the same node together and stop together.

That's useful for a SidecarSidecarA helper process (or container) that runs right next to your app and does one job for it, such as a proxy that handles TLS or a log shipper. They usually talk over a local socket.: a helper container that sits next to your app, such as a log shipper or a proxy. But the rule of thumb is simple: one app container per pod. Your web server and your database go in separate pods, so they can be scaled, upgraded and restarted separately.

Pods are cattle, not pets

Here's the surprising part. Delete that pod, and it's simply gone:

$ kubectl delete pod hello-pod
pod "hello-pod" deleted from default namespace
$ kubectl get pods
No resources found in default namespace.

Nothing brings it back, because nothing was told to keep it alive. And if its node dies, a bare pod dies with it. This is deliberate. Kubernetes treats pods like cattle, not pets: no pod is special, none is nursed back to health, and any one can be replaced by an identical new one, with a new name and a new IP address.

So in practice you never create pods by hand. You create a Deployment, which creates pods for you and replaces them when they vanish. That's the next section.

Deployments and ReplicaSets

The manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 3                 # ← desired state: three pods
  selector:
    matchLabels:
      app: hello              # ← "the pods I own are the ones labelled app=hello"
  template:                   # ← the pod template: what each pod looks like
    metadata:
      labels:
        app: hello            # ← every pod gets this label (must match the selector)
    spec:
      containers:
        - name: hello
          image: hello:v1
          ports:
            - containerPort: 8080

Read it from the bottom up. The pod templatePod templateThe part of a Deployment (under spec.template) that says what each pod should look like: its labels, containers, image, env, probes and resources. Changing it is what triggers a rollout. under template: is exactly the pod from §04, minus the name (pods made from it get generated names). replicas: 3 says how many. And selector is how the Deployment recognises its own pods: by a labelLabelA key=value tag you attach to any Kubernetes object, like app=hello or tier=frontend. Labels don't do anything by themselves; other objects find things by them..

Applying it: three objects for the price of one

$ kubectl apply -f deploy.yaml
deployment.apps/hello created
$ kubectl get deploy,rs,pods
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello   3/3     3            3           5s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-6b85969876   3         3         3       5s

NAME                         READY   STATUS    RESTARTS   AGE
pod/hello-6b85969876-rgpqn   1/1     Running   0          5s
pod/hello-6b85969876-t7fns   1/1     Running   0          5s
pod/hello-6b85969876-z2kjg   1/1     Running   0          5s

We created one object and got five. The Deployment created a ReplicaSetReplicaSetThe object that keeps exactly N identical pods running from one pod template. You rarely create one yourself: a Deployment creates a new ReplicaSet for every version and scales them up and down., and the ReplicaSet created three pods. The names show the family tree: hello → hello-6b85969876 → hello-6b85969876-rgpqn.

That middle number, 6b85969876, is a hash of the pod template. Change the template (say, a new image) and you get a new ReplicaSet with a different hash. That's how rolling updates work: the Deployment shifts pods from the old ReplicaSet to the new one (§11).

Self-healing, for real

Now delete one of the pods, as if it had crashed or its node had died. We watched the pod list while doing it (-w keeps printing changes):

$ kubectl delete pod hello-6b85969876-rgpqn
pod "hello-6b85969876-rgpqn" deleted from default namespace

$ kubectl get pods -w          # (started just before the delete)
NAME                     READY   STATUS              RESTARTS   AGE
hello-6b85969876-rgpqn   1/1     Running             0          11s
hello-6b85969876-t7fns   1/1     Running             0          11s
hello-6b85969876-z2kjg   1/1     Running             0          11s
hello-6b85969876-rgpqn   1/1     Terminating         0          12s
hello-6b85969876-9dr66   0/1     Pending             0          0s
hello-6b85969876-9dr66   0/1     ContainerCreating   0          0s
hello-6b85969876-rgpqn   0/1     Completed           0          13s
hello-6b85969876-9dr66   1/1     Running             0          1s

(Repeated lines trimmed.) In the same second the old pod started Terminating, a new pod, …9dr66, appeared as Pending (waiting for a node), then ContainerCreating, then Running. One second from deletion to replacement, measured on this machine. The cluster's event log says who did what:

$ kubectl get events --sort-by=.lastTimestamp
6s   Normal   Killing            pod/hello-6b85969876-rgpqn    Stopping container hello
6s   Normal   SuccessfulCreate   replicaset/hello-6b85969876   Created pod: hello-6b85969876-9dr66
6s   Normal   Scheduled          pod/hello-6b85969876-9dr66    Successfully assigned default/hello-6b85969876-9dr66 to atlas-k8s-control-plane
5s   Normal   Pulled             pod/hello-6b85969876-9dr66    Container image "hello:v1" already present on machine …
5s   Normal   Started            pod/hello-6b85969876-9dr66    Container started

The ReplicaSet controller saw "want 3, have 2" and created a pod. The scheduler assigned it to a node. The kubelet started it. That's the loop from §03, live.

The reconcile loop

Every controllerController (control loop)A program that runs forever in a loop: look at what you asked for (desired state), look at what exists (actual state), and act to close the gap. Kubernetes is built from dozens of them. in Kubernetes runs the same shape of loop. Here's the ReplicaSet controller, simplified to pseudocode:

for {
	want := rs.Spec.Replicas                     // desired state, from etcd
	have := countPods(matching(rs.Spec.Selector)) // actual state, from the API server
	switch {
	case have < want:
		createPods(want - have) // too few → make more
	case have > want:
		deletePods(have - want) // too many → remove some
	}
	waitForChange() // wake up on the next event (or a periodic resync)
}

Two properties make this robust:

• It's level-based, not event-based. The loop doesn't care why there are 2 pods instead of 3: a crash, a dead node, a person typing kubectl delete. It only looks at the current gap. So it can't "miss" an event: the next pass sees the gap anyway.
• It's IdempotentIdempotentDoing it twice has the same effect as doing it once. “Set balance to 50” is idempotent; “add 50 to balance” is not — so only the first is safe to retry blindly.. Running it again when everything is already right does nothing. So it's safe to run constantly, and safe to restart.

Now try it with the loop made visible. This version logs every pass, including the passes where there's nothing to do. Power off a node and notice that its pods first go Unknown and still count, so the controller waits before replacing them:

Scaling is just changing the number

$ kubectl scale deploy/hello --replicas=5
deployment.apps/hello scaled
$ kubectl get pods -l app=hello        # -l = filter by label
NAME                    READY   STATUS    RESTARTS   AGE
hello-779c6b7b5-5w49p   1/1     Running   0          8s
hello-779c6b7b5-6cnlg   1/1     Running   0          4m48s
hello-779c6b7b5-85zc4   1/1     Running   0          6s
hello-779c6b7b5-9qftf   1/1     Running   0          4m53s
hello-779c6b7b5-fjsqg   1/1     Running   0          6s

(This was later in our session, after an upgrade, which is why the hash differs.) kubectl scale is a shortcut. The better habit is to change replicas: in the YAML file and kubectl apply it again, so the file in git stays the single source of truth. Otherwise the next apply of the old file quietly scales you back down.

Services and DNS: a stable address for moving pods

The problem

In §05 a deleted pod was replaced by …9dr66, with a different IP address. If another service had hard-coded the old pod's IP, its requests would now fail. And with three pods, which IP should a caller pick anyway?

The manifest

apiVersion: v1
kind: Service
metadata:
  name: hello            # ← becomes the DNS name: http://hello
spec:
  selector:
    app: hello           # ← send traffic to pods with this label
  ports:
    - port: 80           # the port callers use
      targetPort: 8080   # the port the container listens on

$ kubectl apply -f svc.yaml
service/hello created
$ kubectl get svc hello
NAME    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
hello   ClusterIP   10.96.116.97   <none>        80/TCP    0s
$ kubectl get endpointslices -l kubernetes.io/service-name=hello
NAME          ADDRESSTYPE   PORTS   ENDPOINTS                          AGE
hello-47dgx   IPv4          8080    10.244.0.7,10.244.0.6,10.244.0.8   0s

Two things appeared:

• A ClusterIPClusterIPThe stable virtual IP address a Service gets, reachable only from inside the cluster. No machine actually owns it; each node's network rules translate it to the IP of one ready pod., 10.96.116.97. This address never changes for the life of the Service.
• An EndpointSliceEndpointSliceAn object Kubernetes keeps up to date listing the IP address of every pod that matches a Service's selector, each marked ready or not ready. kube-proxy and ingress controllers read it and send traffic only to the ready ones.: the current list of pods behind it, each with its IP and a ready: true/false flag. A controller keeps that list up to date as pods come, go, and become ready or unready.

Calling it, for real

We started a throwaway pod with a shell (kubectl run client --image=busybox:1.37 …; busybox is a tiny image of standard Unix tools) and sent 30 requests to http://hello from inside the cluster:

$ kubectl exec client -- sh -c 'for i in $(seq 30); do wget -qO- http://hello; done' | sort | uniq -c
     14 hello from hello-6b85969876-9dr66 (v1)
     10 hello from hello-6b85969876-t7fns (v1)
      6 hello from hello-6b85969876-z2kjg (v1)

Every request went to the same address, and all three pods answered. The split is random, not strict turns: in its default mode, kube-proxy picks a pod at random for each new connection. Over many requests that evens out.

How a Service actually works

Nobody "owns" the ClusterIP. No process listens on it. Instead, on every node, kube-proxykube-proxyA small program on every node that turns Services into network rules (usually iptables or nftables rules in the Linux kernel), so traffic sent to a Service's IP is forwarded to one of its pods. reads the EndpointSlices and writes kernel network rules for the ready entries only (iptables or nftables) that say: "a packet to 10.96.116.97:80? Rewrite its destination to one of 10.244.0.6:8080, 10.244.0.7:8080, 10.244.0.8:8080." The rewrite happens in the Linux kernel of whichever node the caller is on, so there's no extra hop through a central Load balancerLoad balancerA component that spreads incoming requests or connections across several servers, so no single one is overloaded and a dead one can be skipped..

Play with it. Change pod labels and readiness and watch where requests can go:

The rule to remember: a pod only receives Service traffic if its labels match the selector and it's ready. An unready pod still appears in kubectl get endpointslices, marked ready: false; it just gets no traffic. "Ready" is decided by its readiness probe (§09). That's how a pod that's still starting, or shutting down, stays out of rotation.

DNS: calling by name

You don't type ClusterIPs either. Every cluster runs cluster DNSCluster DNS (CoreDNS)The DNS server inside every cluster. It answers names like hello.default.svc.cluster.local with the Service's ClusterIP, so pods can call each other by Service name. (CoreDNS, the coredns pods from §03), and every Service gets a name:

$ kubectl exec client -- nslookup hello.default.svc.cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53

Name:	hello.default.svc.cluster.local
Address: 10.96.116.97

The full name is <service>.<namespace>.svc.cluster.local. You rarely type all of it, thanks to the pod's DNS settings:

$ kubectl exec client -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

The search line means a short name like hello is tried as hello.default.svc.cluster.local first. So from the same namespace, http://hello just works; from another namespace, use http://hello.<namespace> (§12). The ndots:5 line has a performance surprise for external names; DNS §08 explains it.

Types of Service

For HTTP apps with many services, you usually don't give each one a LoadBalancer. You put one entry point in front of all of them: that's Ingress, next.

Reaching a Service from your laptop

While developing or debugging, port-forwardkubectl port-forwardA command that opens a tunnel from a port on your laptop to a pod or Service inside the cluster, through the API server. Handy for debugging; not a way to serve real traffic. opens a tunnel from your machine into the cluster:

$ kubectl port-forward svc/hello 8081:80 &
$ curl localhost:8081
hello from hello-6b85969876-t7fns (v1)

The tunnel goes through the API server and connects to one pod picked when it starts (no spreading across pods), so it's slow and meant for one person. It isn't how real users reach your app.

Ingress: letting the outside world in

Services from §06 handle traffic inside the cluster. Real users arrive from the internet, usually over HTTPS, often for many different apps. You don't want one paid cloud load balancer per Service. You want one front door that looks at each request's hostname and path and forwards it to the right Service.

The manifest

Illustrative: our kind cluster had no ingress controller installed, so we checked this file with kubectl apply --dry-run=server (the API server validated it and said ingress.networking.k8s.io/shop created (server dry run)) but didn't send real traffic through it.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
spec:
  ingressClassName: nginx       # ← which installed controller should handle this
  tls:
    - hosts: [shop.example.com]
      secretName: shop-tls      # a Secret holding the certificate and key (§08)
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix    # /api, /api/, /api/cart… but not /apiary
            backend:
              service:
                name: api
                port:
                  number: 80
          - path: /
            pathType: Prefix    # everything else
            backend:
              service:
                name: web
                port:
                  number: 80

Three things the controller does with this:

1. Matches the host. Requests for other hostnames don't use these rules; if nothing matches at all, they go to the controller's default backendDefault backendWhere an ingress controller sends a request that matches none of its rules: its fallback, which usually answers 404 Not Found..
2. Matches the path. With Prefix, a path matches when it starts with the rule's path element by element, and the longest matching rule wins. So /api/cart goes to api even though / also matches.
3. Terminates TLSTLSTransport Layer Security: the encryption layer that makes HTTPS secure. It proves the server is who it claims to be and scrambles everything so nobody in between can read or change it.. It decrypts HTTPS using the certificate in shop-tls, then talks plain HTTP to the pods inside the cluster.

Try some URLs against those rules:

No controller, no traffic

The most common Ingress surprise: you apply an Ingress, kubectl get ingress shows it, and nothing happens. Kubernetes doesn't ship an ingress controllerIngress controllerThe actual reverse proxy (NGINX, Traefik, HAProxy, a cloud load balancer…) that reads Ingress objects and routes outside HTTP traffic to Services. Without one installed, an Ingress does nothing.. The Ingress object is only a set of rules; you must install a controller (Traefik, HAProxy, Contour, your cloud's own…) that reads them. ingressClassName says which one should pick this Ingress up. Managed clusters usually offer one as an add-on.

The successor: Gateway API

Ingress is deliberately small: host and path routing, TLS, and not much else. Everything beyond that (redirects, header matching, splitting traffic 90/10 between two versions) was done with controller-specific annotationsAnnotationA key/value note attached to a Kubernetes object, for tools and humans: who changed it, which controller settings to use, a checksum. Unlike a label, nothing selects objects by annotations. that differed between controllers. The Gateway APIGateway APIThe newer, more expressive Kubernetes standard for routing outside traffic into the cluster (Gateway + HTTPRoute objects). It's the successor to Ingress, with features like traffic splitting built in. is the newer standard that builds those features in. It splits the job into a Gateway (the entry point, owned by the platform team) and HTTPRoutes (routing rules, owned by each app team):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
spec:
  parentRefs:
    - name: public-gateway      # attach to the shared Gateway
  hostnames: ["shop.example.com"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /api }
      backendRefs:
        - name: api
          port: 80
    - backendRefs:              # no matches = everything else
        - name: web
          port: 80

ConfigMaps and Secrets: configuration outside the image

Containers §08 made the case for keeping config out of images: the image you tested in Staging environmentStaging environmentA copy of production — same code, similar machines and data — used to test changes before they reach real users. Load tests usually run here. should be byte-for-byte the image you run in production. Kubernetes gives that config a home in the cluster.

Two objects

apiVersion: v1
kind: ConfigMap
metadata:
  name: hello-config
data:                          # plain key/value settings
  GREETING: "hi"
  LOG_LEVEL: "info"
---
apiVersion: v1
kind: Secret
metadata:
  name: hello-secret
type: Opaque                   # "arbitrary data" (other types: TLS certs, registry logins)
stringData:                    # write plain text here; Kubernetes stores it encoded
  DB_PASSWORD: "s3cr3t-dev-only"

A ConfigMapConfigMapA Kubernetes object holding non-secret settings that pods read as environment variables or files. is for anything you'd be happy to see in a log. A SecretSecret (Kubernetes)A Kubernetes object for passwords and keys, given to pods as env vars or files. Only base64-encoded by default, so access control and encryption at rest matter. is for passwords, API keys and certificates (the Ingress in §07 read its TLS certificate from one). The difference is mostly in who may read them, as you'll see.

Handing them to the container

The usual way is as environment variablesEnvironment variableA named setting the operating system hands a program when it starts, like DATABASE_URL=postgres://… Read in Go with os.Getenv and in Node with process.env.. This goes inside the container in the Deployment's pod template:

      containers:
        - name: hello
          image: hello:v2
          envFrom:
            - configMapRef:
                name: hello-config     # ← every key becomes an env var
          env:
            - name: DB_PASSWORD        # ← one env var, taken from one Secret key
              valueFrom:
                secretKeyRef:
                  name: hello-secret
                  key: DB_PASSWORD

The app just reads its environment, exactly as it would on your laptop:

greeting := os.Getenv("GREETING") // "hi" — from the ConfigMap
if greeting == "" {
	greeting = "hello" // a default, so the app also runs without Kubernetes
}
dbPass := os.Getenv("DB_PASSWORD") // from the Secret

After the update, our app greeted differently, with no image rebuild:

$ kubectl exec client -- wget -qO- http://hello
hi from hello-779c6b7b5-9qftf (v2)

And kubectl describe pod shows where each value came from, without printing the secret itself:

    Environment Variables from:
      hello-config  ConfigMap  Optional: false
    Environment:
      DB_PASSWORD:  <set to the key 'DB_PASSWORD' in secret 'hello-secret'>  Optional: false

Or as files

Some apps want a config file. Mount the ConfigMap as a volumeVolume (Kubernetes)A folder mounted into a pod's containers. Its contents can come from a ConfigMap, a Secret, a persistent disk, or an empty scratch directory that lives as long as the pod. (a folder the pod mounts) and each key becomes a file:

    spec:
      containers:
        - name: hello
          image: hello:v2
          volumeMounts:
            - name: config
              mountPath: /etc/hello    # → /etc/hello/GREETING, /etc/hello/LOG_LEVEL
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: hello-config

Changing config: the trap

Edit the ConfigMap and apply it. What happens to running pods?

• Env vars don't change. A process's environment is fixed when it starts. The pods keep the old values until they're replaced.
• Mounted files do change, after a delay. Your app must notice and re-read the file, which most don't. (Exception: a file mounted with subPath never updates.)

We tested both at once: a pod that prints its env var and its mounted file every 5 seconds. We changed GREETING from hi to hey at 23:33:23. kubectl patch edits fields of an object in place, without a YAML file:

$ kubectl patch configmap hello-config -p '{"data":{"GREETING":"hey"}}'
configmap/hello-config patched
$ kubectl logs cmwatch
23:33:23 env=hi file=hi
…
23:34:43 env=hi file=hi
23:34:48 env=hi file=hey      ← the file caught up after ~85 s
23:34:53 env=hi file=hey      ← the env var never will

So after changing config, restart the pods deliberately. This does a rolling restart (§11), with no downtime:

$ kubectl rollout restart deploy/hello
deployment.apps/hello restarted

A common pattern is to put a version in the ConfigMap's name (hello-config-v7) and point the Deployment at the new name. Then changing config is a change to the pod template, which triggers a normal rollout you can undo. Kustomize (§12) can generate those names for you, adding a hash of the contents; Helm charts often get the same effect by putting a checksum of the config into the pod template.

Secrets are not encrypted (by default)

Here's what the Secret looks like when you read it back:

$ kubectl get secret hello-secret -o yaml
data:
  DB_PASSWORD: czNjcjN0LWRldi1vbmx5
$ kubectl get secret hello-secret -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
s3cr3t-dev-only

That's Base64Base64A way to write any bytes using only 64 safe text characters (A–Z, a–z, 0–9, +, /). Every 3 bytes become 4 characters, so binary data can travel inside text like HTTP headers.: an encoding, not encryption. Anyone who can read the Secret object can decode it in one command. What actually protects Secrets is:

1. Access control. RBACRBAC (role-based access control)Granting permissions to roles (admin, editor, viewer) and roles to users, instead of granting each permission to each user directly. rules decide who may get Secrets in each namespace. Many people can see Deployments; very few should see Secrets.
2. Encryption at rest. The cluster can be configured to encrypt Secrets inside etcd. Managed clusters usually do this, or offer it as a setting. Check yours.
3. Not putting them in git. A Secret manifest with stringData is a plaintext password. Don't commit it.

In production, teams usually keep the real values in a secrets manager (Vault, AWS Secrets Manager…) and sync them into the cluster with a tool like External Secrets Operator. Secrets §10 covers those options and their trade-offs.

Probes: liveness, readiness and startup

Load balancing §06 explains health checks in general, and why liveness and readiness must check different things. This section is the Kubernetes version: how to write the probes, and what really happens when they fail.

The app's side: two endpoints

Our app gets two health endpoints. /healthz answers if the process can answer at all. /readyz answers 200 only after a 3-second warm-up (pretend it's loading a cache), and 503 before that:

mux := http.NewServeMux()   // a router: maps URL paths to handler functions
var ready atomic.Bool       // false until warm-up is done

mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte("ok")) // liveness: "I can answer". No database checks!
})
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
	if !ready.Load() {
		http.Error(w, "warming up", http.StatusServiceUnavailable) // 503 → no traffic yet
		return
	}
	w.Write([]byte("ready"))
})

go func() {
	time.Sleep(3 * time.Second) // pretend to load caches
	ready.Store(true)
}()

The cluster's side: the probe config

          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 2        # ask every 2 s
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 10       # ask every 10 s…
            failureThreshold: 3     # …restart after 3 failures in a row (~30 s)

A probe passes when the HTTP status is 200–399 within timeoutSeconds (default 1 s). Besides httpGet, there are tcpSocket (can I connect?), grpc (the standard health check for gRPCgRPCA popular framework for one service to call functions on another service over the network. It passes deadlines along automatically. servers) and exec (run a command in the container; exit code 0 = pass).

kubectl describe pod prints the full settings, including the defaults we didn't write:

    Liveness:   http-get http://:8080/healthz delay=0s timeout=1s period=10s successThreshold=1 failureThreshold=3
    Readiness:  http-get http://:8080/readyz delay=0s timeout=1s period=2s successThreshold=1 failureThreshold=3

delay is initialDelaySeconds (wait before the first probe), timeout is how long one probe may take, and successThreshold=1 means one pass is enough to count as healthy again.

Play with the three probes on one pod. Pick a scenario, press play, and watch what the kubelet does:

Readiness failing, for real

We shipped a broken version, v3-broken, whose /readyz always returns 503 ("cannot reach database"). The pod starts fine but never becomes ready:

$ kubectl get pods -l app=hello
NAME                    READY   STATUS    RESTARTS   AGE
hello-779c6b7b5-6cnlg   1/1     Running   0          35s
hello-779c6b7b5-9qftf   1/1     Running   0          40s
hello-779c6b7b5-zfrbh   1/1     Running   0          45s
hello-f8db94ffd-q5vqw   0/1     Running   0          20s

Running but 0/1 READY: the process is up, the probe says "don't send me traffic". describe shows why:

$ kubectl describe pod hello-f8db94ffd-q5vqw
…
Events:
  Type     Reason     Age                From     Message
  Normal   Started    20s                kubelet  spec.containers{hello}: Container started
  Warning  Unhealthy  1s (x11 over 20s)  kubelet  spec.containers{hello}: Readiness probe failed: HTTP probe failed with statuscode: 503

And users? We sent requests to the Service while this pod existed. Every one went to the healthy v2 pods; the unready pod got nothing:

$ kubectl exec client -- sh -c 'for i in 1 2 3 4 5; do wget -qO- http://hello; done'
hi from hello-779c6b7b5-zfrbh (v2)
hi from hello-779c6b7b5-9qftf (v2)
hi from hello-779c6b7b5-zfrbh (v2)
hi from hello-779c6b7b5-9qftf (v2)
hi from hello-779c6b7b5-6cnlg (v2)

Readiness also stopped the rollout from going any further: no old pod was removed, because the new one never became ready. That's §11's safety net.

Liveness failing, for real

Now a typo: a liveness probe pointing at port 8081 when the app listens on 8080. The app is perfectly healthy, but the kubelet can't reach it. This pod, badlive, had this probe:

      livenessProbe:
        httpGet:
          path: /healthz
          port: 8081          # typo: the app listens on 8080
        periodSeconds: 5
        failureThreshold: 3

$ kubectl get pod badlive
NAME      READY   STATUS    RESTARTS     AGE
badlive   1/1     Running   3 (4s ago)   50s
$ kubectl describe pod badlive
…
  Warning  Unhealthy  4s (x9 over 44s)  kubelet  Liveness probe failed: Get "http://10.244.0.17:8081/healthz": dial tcp 10.244.0.17:8081: connect: connection refused
  Normal   Killing    4s (x3 over 34s)  kubelet  Container hello failed liveness probe, will be restarted

Three restarts in 50 seconds (5 s period × 3 failures ≈ one restart every 15 s). Every restart throws away in-flight requests and warm caches. A wrong liveness probe is worse than none.

Startup probes: for slow starters

Some apps take a long time to start (a big Java app warming up, a large cache load, database migrations). With only a liveness probe, you'd have to pick: a long initialDelaySecondsinitialDelaySecondsA probe setting: how long the kubelet waits after the container starts before running the first probe. Default 0. (wait that long before the first probe; slow to catch real hangs forever after), or a short one (the app is killed while still starting, restarts, gets killed again…).

A startup probeStartup probeA Kubernetes health check that runs only while a container is starting. Until it succeeds, liveness and readiness checks are paused — so a slow-starting app isn't restarted for being slow. fixes that. Until it succeeds, liveness and readiness are paused:

          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 5
            failureThreshold: 24    # 24 × 5 s = up to 2 minutes to start
          livenessProbe:            # only starts once the startup probe has passed
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 10

What each probe should check

Requests, limits and OOMKilled

The settings

          resources:
            requests:
              cpu: 50m          # reserve 5% of one CPU core
              memory: 32Mi      # reserve 32 MiB of RAM
            limits:
              memory: 64Mi      # hard ceiling: above this, the container is killed

These are the requests and limitsRequests and limitsPer-container resource settings. A request is what the scheduler reserves for the container on a node (and its guaranteed share). A limit is the hard ceiling: over the memory limit it's killed, at the CPU limit it's slowed down. we ran with. (Our demo deliberately sets the memory request below the limit, to show the QoS class further down; the staff tip at the end explains why you'd usually make them equal.)

Requests: where a pod fits

The schedulerScheduler (kube-scheduler)The control-plane component that picks a node for every new pod, based on how much CPU and memory the pod asks for and what each node has free. It only decides; the node's kubelet starts the pod. only looks at requests, never at actual usage. A node can host a pod only if the sum of requests already on it, plus the new pod's request, fits in the node's allocatable capacity. Here's our node:

$ kubectl describe node
Allocatable:
  cpu:                4
  memory:             16477352Ki
  pods:               110
…
Allocated resources:
  Resource           Requests     Limits
  cpu                1100m (27%)  0 (0%)
  memory             386Mi (2%)   532Mi (3%)

So what happens if we ask for 64 CPUs on a 4-CPU node?

$ kubectl get pod greedy
NAME     READY   STATUS    RESTARTS   AGE
greedy   0/1     Pending   0          4s
$ kubectl describe pod greedy
Events:
  Type     Reason            Age              From               Message
  Warning  FailedScheduling  4s (x3 over 4s)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

(The second half is about preemptionPreemption (scheduler)When no node has room for a pod, the scheduler may evict lower-priority pods to make space for a higher-priority one. With equal priorities (the default) it can't help, as the message says.: making room by evicting lower-priority pods. All our pods have the same priority, so that can't help.)

Pending means "no node has room". It waits forever, or until a node frees up, or until the cluster autoscaler (§12) adds a node. The message tells you which resource is missing.

Limits: the ceiling

The two kinds of limit fail very differently, because memory and CPU are different kinds of resource:

These limits are enforced by Linux cgroupscgroupsControl groups: a Linux kernel feature that limits how much CPU, memory and how many processes a group of processes may use. It controls how much a container can use., exactly as with docker run -m in Containers.

OOMKilled, for real

Our app has a /hog?mb=N endpoint that allocates N MB and keeps it. The container's memory limit is 64 Mi. We asked one pod for 100 MB:

$ kubectl exec client -- wget -qO- "http://10.244.0.12:8080/hog?mb=100"
wget: error getting response
$ kubectl get pods -l app=hello
NAME                    READY   STATUS    RESTARTS     AGE
hello-779c6b7b5-6cnlg   0/1     Running   1 (3s ago)   54s
…
$ kubectl describe pod hello-779c6b7b5-6cnlg
    State:          Running
      Started:      Fri, 30 Oct 2026 22:58:30 +0000
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 30 Oct 2026 22:57:38 +0000
      Finished:     Fri, 30 Oct 2026 22:58:29 +0000
    Restart Count:  1

The kernel killed the process mid-request (so wget got no response). The kubelet restarted the container in the same pod (RESTARTS 1), and it was briefly 0/1 while warming up again. Exit codeExit codeA small number a process returns when it ends: 0 means success, anything else means failure. 128 + N means it was killed by signal N (137 = SIGKILL, 139 = segfault). 137 = 128 + 9: killed by SIGKILL. The current state looks fine; the clue is in Last State. That's why "it keeps restarting" investigations start with describe.

Try it yourself: place pods on two nodes, then push one pod's memory use past its limit:

Tell your runtime about the limit

Runtimes with a garbage collectorGarbage collector (GC)Part of the language runtime that automatically frees memory nobody can reach anymore. Anything still referenced — like a stuck goroutine — is never freed. decide how much memory to use before cleaning up. If their idea of "how much is allowed" is bigger than the container's limit, they grow straight past it and get killed, when collecting garbage a bit earlier would have been fine. Go doesn't look at the limit at all by default. Node (20 and newer) does size its default heap from the limit, but never below roughly 256 MB, so under a small limit like our 64Mi it will happily overshoot:

# Go: a soft memory target. The GC works harder as the heap nears it.
ENV GOMEMLIMIT=56MiB          # ~90% of a 64Mi limit, leaving headroom

You can also set GOMEMLIMIT from the pod spec, so it follows the limit automatically. The resourceFieldRef feature fills an env var with the container's own limit. Note that this gives Go 100% of the limit, with no headroom for memory outside Go's heap; if you see OOM kills anyway, switch to a fixed value around 90% as above:

          env:
            - name: GOMEMLIMIT
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory   # e.g. 67108864 (bytes); Go accepts a plain number

For CPU, Go 1.25 and newer set GOMAXPROCS from the container's CPU limit automatically; older Go versions see every core on the node.

QoS classes: who goes first when memory runs out

From your requests and limits, Kubernetes puts each pod in a QoS classQoS classA label Kubernetes gives each pod from its requests and limits: Guaranteed (requests = limits), Burstable (some requests) or BestEffort (none). When a node runs short of memory, BestEffort pods are evicted first. (kubectl describe showed ours as QoS Class: Burstable):

That order is a simplification. When a node runs short, the kubelet evictsEvictionKubernetes removing a pod from a node on purpose — because the node is being drained for maintenance, or because the node is running out of memory or disk. The pod's controller then creates a replacement elsewhere. pods mainly by how far their usage is above their request (and by priority), and the QoS class mostly decides who the kernel's OOM killer picks first. In practice it works out roughly as the table says.

Rolling updates and rollbacks

Deployment strategies compares rolling updates with blue-green and canary releases. Here we look at how Kubernetes does the rolling kind, which is the Deployment default.

Triggering one

We changed image: hello:v1 to image: hello:v2 in the YAML (along with the probes and config from §08–§10) and applied it:

$ kubectl apply -f deploy.yaml
deployment.apps/hello configured
$ kubectl rollout status deploy/hello
Waiting for deployment "hello" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "hello" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello" rollout to finish: 1 old replicas are pending termination...
deployment "hello" successfully rolled out

(Repeated lines trimmed.) 16 seconds from apply to done, measured on this machine. Here's the pod list during that time, trimmed to the moments that matter:

$ kubectl get pods -l app=hello -w
hello-6b85969876-9dr66   1/1     Running             0          52s    ← three v1 pods
hello-6b85969876-t7fns   1/1     Running             0          64s
hello-6b85969876-z2kjg   1/1     Running             0          64s
hello-779c6b7b5-zfrbh    0/1     ContainerCreating   0          0s     ← v2 pod #1 starts (4 pods now)
hello-779c6b7b5-zfrbh    0/1     Running             0          0s     ← running, not ready yet
hello-779c6b7b5-zfrbh    1/1     Running             0          5s     ← ready after warm-up
hello-6b85969876-9dr66   1/1     Terminating         0          59s    ← only now: one v1 pod stops
hello-779c6b7b5-9qftf    0/1     Pending             0          0s     ← v2 pod #2
hello-779c6b7b5-9qftf    1/1     Running             0          5s
hello-6b85969876-t7fns   1/1     Terminating         0          76s
hello-779c6b7b5-6cnlg    1/1     Running             0          5s     ← v2 pod #3 ready
hello-6b85969876-z2kjg   1/1     Terminating         0          81s    ← last v1 pod stops

Notice the order: a v1 pod was only stopped after a v2 pod turned ready. The readiness probe from §09 is what makes this safe. Without one, Kubernetes would treat "process started" as "ready" and could send users to an app that's still loading.

Under the hood, the Deployment made a second ReplicaSet and moved the count across:

$ kubectl get rs
NAME               DESIRED   CURRENT   READY   AGE
hello-6b85969876   0         0         0       83s    ← v1, scaled to 0 but kept
hello-779c6b7b5    3         3         3       17s    ← v2

The old ReplicaSet is kept at 0 replicas on purpose: it's the recipe for rolling back.

The two knobs

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1          # may run 1 pod above 3 during the update (so up to 4)
      maxUnavailable: 0    # must never drop below 3 ready pods

maxSurge and maxUnavailablemaxSurge / maxUnavailableThe two knobs of a rolling update. maxSurge: how many extra pods above the desired count may exist during the update. maxUnavailable: how many of the desired pods may be missing or not ready. set the pace. Each can be a number or a percentage of replicas, and both default to 25%:

• maxSurge: 1, maxUnavailable: 0 (what we used): never lose capacity, but needs room for one extra pod. Safest, slowest.
• maxSurge: 0, maxUnavailable: 1: no extra pods (good when the cluster is full), but you run one short during the update.
• Bigger numbers go faster and take bigger risks.

The other strategy, type: Recreate, kills all old pods first and then starts the new ones. That means downtime, but it's sometimes needed when two versions can't run side by side (say, both would migrate the same database).

Try the knobs, and a broken version:

A broken release, for real

Now v3-broken, whose readiness probe always fails (§09). We used kubectl set image, a shortcut that edits the image in place:

$ kubectl set image deploy/hello hello=hello:v3-broken
deployment.apps/hello image updated
$ kubectl rollout status deploy/hello --timeout=20s
Waiting for deployment "hello" rollout to finish: 1 out of 3 new replicas have been updated...
error: timed out waiting for the condition
$ kubectl get deploy hello
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
hello   3/3     1            3           116s

Read get deploy carefully: 3/3 ready and 3 available (the old v2 pods, still serving), and 1 up-to-date (the new v3 pod, never ready). With maxUnavailable: 0, Kubernetes won't remove a v2 pod until a v3 pod is ready, and that never happens. The rollout is stuck, and users don't notice.

How long until Kubernetes calls it failed? That's the progress deadlineProgress deadlineHow long a Deployment may go without making progress (default 600 s) before Kubernetes marks the rollout as failed (ProgressDeadlineExceeded). It reports the failure; it doesn't roll back by itself., progressDeadlineSeconds, 600 s by default. We set it to 20 s to see it:

$ kubectl rollout status deploy/hello
Waiting for deployment "hello" rollout to finish: 1 out of 3 new replicas have been updated...
error: deployment "hello" exceeded its progress deadline
$ kubectl get deploy hello -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
Available=True MinimumReplicasAvailable
Progressing=False ProgressDeadlineExceeded

Kubernetes reports the failure (and rollout status exits with an error, which fails a CI job), but it does not roll back by itself. The broken pod stays there until you act. Automatic rollback based on health or error rate needs extra tooling, covered in Deployment strategies.

Rolling back

$ kubectl rollout undo deploy/hello
deployment.apps/hello rolled back
$ kubectl rollout status deploy/hello
deployment "hello" successfully rolled out
$ kubectl rollout history deploy/hello
REVISION  CHANGE-CAUSE
1         <none>
3         <none>
4         <none>

undo scaled the v2 ReplicaSet back up and the v3 one to zero. It's a rollout like any other, just towards an older template. In the history, revision 2 (v2) was re-used as revision 4; that's why it's missing. (CHANGE-CAUSE stays empty unless you set the kubernetes.io/change-cause annotationAnnotationA key/value note attached to a Kubernetes object, for tools and humans: who changed it, which controller settings to use, a checksum. Unlike a label, nothing selects objects by annotations. (a free-form note on the object), which some CI pipelines do.)

undo also printed a warning, which points at a real trap:

Warning: resource deployments/hello was previously managed with 'kubectl apply'. Rolling back will not
update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior
on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.

The cluster now runs v2, but your YAML file (and git) still say v3. The next kubectl apply of that file brings v3 right back. rollout undo is for emergencies; the real fix is reverting the change in git and applying again. With a GitOpsGitOpsRunning deployments by committing the desired configuration (Kubernetes manifests) to a git repo and letting a tool apply whatever is in git to the cluster. tool, the git revert is the rollback.

Autoscaling, namespaces and packaging YAML

Autoscaling pods: the HPA

In §05 we scaled by changing replicas ourselves. A HorizontalPodAutoscalerHorizontalPodAutoscaler (HPA)An object that changes a Deployment's replica count automatically, aiming to keep a metric (usually average CPU use as a percent of requests) near a target. changes it for you, aiming to keep a metric near a target. ("Horizontal" means more copies, as opposed to bigger ones.) Load balancing discusses AutoscalingAutoscalingAutomatically adding instances when a metric (CPU, requests per second, latency) is high and removing them when it's low. in general; this is the Kubernetes version.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: hello
spec:
  scaleTargetRef:              # which Deployment to scale
    apiVersion: apps/v1
    kind: Deployment
    name: hello
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70   # aim for 70% of the pods' CPU *request*

The arithmetic, with real numbers. Every 15 seconds the HPA computes:

Say 3 pods each request 200m CPU and are using 140m on average. That's 70% of their request, exactly on target: ceil(3 × 70 ÷ 70) = 3. Traffic doubles, usage goes to 280m = 140%: ceil(3 × 140 ÷ 70) = 6. The HPA sets replicas: 6. When traffic drops, it scales back down, but slowly: by default it waits for 5 minutes of lower readings first, so a short dip doesn't throw away pods you'll need again a minute later.

Notice the target is a percentage of the request (§10). Pods without a CPU request can't be autoscaled on CPU.

It needs metrics, for real

We created the HPA with the shortcut command (--cpu=70% in kubectl 1.37; older versions spell it --cpu-percent=70), and here's what happened on our fresh kind cluster:

$ kubectl autoscale deploy hello --cpu=70% --min=3 --max=10
horizontalpodautoscaler.autoscaling/hello autoscaled
$ kubectl get hpa
NAME    REFERENCE          TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
hello   Deployment/hello   cpu: <unknown>/70%   3         10        3          20s
$ kubectl describe hpa hello
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get
  cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the
  server could not find the requested resource (get pods.metrics.k8s.io)

<unknown>: the HPA can't see CPU usage, because nothing is collecting it. It needs metrics-servermetrics-serverA small add-on that collects current CPU and memory use from every kubelet. The HPA and kubectl top need it, and it isn't installed in every cluster by default., an add-on that kind doesn't install (most managed clusters do). If your HPA never scales, check TARGETS first. Scaling on other metrics, like requests per second or queue length, needs a further adapter (KEDA is a popular one).

Namespaces: folders in one cluster

A namespaceNamespace (Kubernetes)A named folder for objects inside one Kubernetes cluster (e.g. billing-prod). Permissions and Secrets are scoped to it: a pod can only use Secrets from its own namespace. is a named folder for objects. Names only have to be unique within one namespace, so two teams can both have a Deployment called hello:

$ kubectl create namespace staging
namespace/staging created
$ kubectl -n staging create deployment hello --image=hello:v1 --port=8080
$ kubectl -n staging expose deployment hello --port=80 --target-port=8080
$ kubectl get pods                       # the "default" namespace
NAME                    READY   STATUS    RESTARTS        AGE
client                  1/1     Running   0               5m1s
hello-779c6b7b5-6cnlg   1/1     Running   1 (3m37s ago)   4m28s
hello-779c6b7b5-9qftf   1/1     Running   0               4m33s
hello-779c6b7b5-zfrbh   1/1     Running   0               4m38s
$ kubectl get pods -n staging
NAME                     READY   STATUS    RESTARTS   AGE
hello-6b85969876-dm2wl   1/1     Running   0          1s

Two Services named hello, in different namespaces. DNS tells them apart by the namespace part of the name (§06):

$ kubectl exec client -- sh -c 'wget -qO- http://hello; wget -qO- http://hello.staging'
hi from hello-779c6b7b5-zfrbh (v2)
hello from hello-6b85969876-dm2wl (v1)

Namespaces are where teams draw lines:

• Permissions. RBACRBAC (role-based access control)Granting permissions to roles (admin, editor, viewer) and roles to users, instead of granting each permission to each user directly. rules are usually per namespace: the payments team can deploy to payments, and not to search.
• Budgets. A ResourceQuotaResourceQuotaA per-namespace budget: the total CPU, memory and number of objects that everything in that namespace may request. Stops one team's namespace from eating the whole cluster. caps what a namespace can request in total:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-budget
  namespace: staging
spec:
  hard:
    requests.cpu: "4"        # all pods in staging together may request at most 4 cores
    requests.memory: 8Gi
    pods: "40"

Packaging all this YAML

A real app has a Deployment, a Service, an Ingress, a ConfigMap, an HPA… times three environments (dev, staging, prod) that differ in a few values. Copy-pasting those files gets out of sync quickly. Two tools dominate.

Kustomize: a base plus patches

KustomizeKustomizeA tool built into kubectl (kubectl apply -k) that builds variants of plain YAML by layering patches — e.g. a base Deployment plus a “prod” overlay that changes replicas and the image tag. is built into kubectl. You keep one plain-YAML base and small overlays that change only what differs:

base/
  deploy.yaml            ← the Deployment from §05: replicas: 1, image: hello:v1,
  kustomization.yaml        envFrom: configMapRef: hello-config
overlays/prod/
  kustomization.yaml     ← below

resources:
  - deploy.yaml

resources:
  - ../../base          # start from the shared base
namespace: prod         # put everything in the prod namespace
images:
  - name: hello
    newTag: v2          # hello:v1 → hello:v2
replicas:
  - name: hello
    count: 5            # 1 → 5
configMapGenerator:
  - name: hello-config  # generated, with a hash of its contents in the name
    literals:
      - GREETING=hi

kubectl kustomize shows the result without applying it (kubectl apply -k applies it). Real output, trimmed:

$ kubectl kustomize overlays/prod
apiVersion: v1
kind: ConfigMap
metadata:
  name: hello-config-hc7d4825hb       ← hash of the contents
  namespace: prod
…
kind: Deployment
metadata:
  name: hello
  namespace: prod
spec:
  replicas: 5
  …
      containers:
      - envFrom:
        - configMapRef:
            name: hello-config-hc7d4825hb   ← reference rewritten to match
        image: hello:v2

That hashed name is the trick from §08: change the config, and the name changes, so the pod template changes, so a normal rollout restarts the pods.

Helm: templates plus values

HelmHelmA package manager for Kubernetes: a chart is a set of templated YAML files you install with your own values. takes the other approach: YAML templates with placeholders, filled in from a values file. A chartHelm chart and valuesA chart is a folder of templated Kubernetes YAML plus a values.yaml of defaults. Installing it with your own values (image tag, replicas…) renders the templates into real manifests; one installed copy is a release. is a folder of these. helm create makes a starter chart (we ran Helm 4.3):

$ helm create hello
Creating hello
$ grep -n "replicas\|image:" hello/templates/deployment.yaml
9:  replicas: {{ .Values.replicaCount }}
41:          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
$ helm template web ./hello --set replicaCount=3 --set image.repository=hello --set image.tag=v2 | grep -E "replicas:|image:"
  replicas: 3
          image: "hello:v2"
      image: busybox

(The last line comes from the starter chart's test pod.) helm template renders without installing; helm install web ./hello -f prod-values.yaml renders and applies, and helm upgrade / helm rollback manage versions of that installed release. The real draw is the ecosystem: databases, monitoring stacks and ingress controllers all ship as charts you can install with your own values.

Debugging with kubectl

Step 1: get — read the STATUS column

$ kubectl get pods
NAME      READY   STATUS             RESTARTS      AGE
crashy    0/1     Error              4 (58s ago)   98s
typo      0/1     ImagePullBackOff   0             98s
badlive   0/1     CrashLoopBackOff   4 (2s ago)    98s
greedy    0/1     Pending            0             97s

Four broken pods we created on purpose, each broken a different way. The STATUS column, READY and RESTARTS already tell you which part of the pod's life went wrong:

Step 2: describe — the Events at the bottom

kubectl describe pod <name> prints everything about the pod. The most useful part is at the very end: Events, a log of what the scheduler and kubelet did to it. For the typo pod (image hello:v9, which doesn't exist):

$ kubectl describe pod typo
Events:
  Type     Reason     Age                From     Message
  Normal   Scheduled  76s                default-scheduler  Successfully assigned default/typo to atlas-k8s-control-plane
  Normal   Pulling    37s (x3 over 75s)  kubelet  Pulling image "hello:v9"
  Warning  Failed     37s (x3 over 75s)  kubelet  Failed to pull image "hello:v9": … failed to resolve reference "docker.io/library/hello:v9" …
  Warning  Failed     37s (x3 over 75s)  kubelet  Error: ErrImagePull
  Normal   BackOff    12s (x4 over 75s)  kubelet  Back-off pulling image "hello:v9"
  Warning  Failed     12s (x4 over 75s)  kubelet  Error: ImagePullBackOff

Notice docker.io/library/hello:v9: a short image name with no registry means Docker Hub. When an image "works on my laptop" but not in the cluster, it's often because your laptop had it locally and the cluster has to download it.

Events are kept for about an hour by default, so check them soon after the problem.

Step 3: logs — what the app said

$ kubectl logs crashy
config file missing
$ kubectl logs deploy/hello            # picks one pod of the Deployment
Found 3 pods, using pod/hello-779c6b7b5-zfrbh
2026/10/30 22:57:28 v2 listening on :8080
2026/10/30 22:57:31 ready

Useful flags:

• --previous (-p): logs of the last crashed container. For a crash loop, the current container may have only just started (or not yet), so the error is in the previous one.
• -f: keep streaming new lines, like tail -f.
• -c <name>: pick the container, if the pod has more than one.
• -l app=hello --prefix: logs from all pods with that label, each line tagged with its pod.

The kubelet keeps only the current and the previous container's logs, on that node. Once a pod is deleted its logs are gone, which is why production clusters ship logs to a central store (Observability).

Step 4: exec and debug — look inside

kubectl exec -it <pod> -- sh opens a shell in a running container, if the image has a shell. Ours doesn't: it's built FROM scratch, with nothing but the Go binary.

$ kubectl exec hello-779c6b7b5-9qftf -- sh
error: … exec: "sh": executable file not found in $PATH

That's good for security (an attacker gets no tools either), but how do you debug it? kubectl debug adds an ephemeral debug containerEphemeral debug containerA temporary container that kubectl debug adds to a running pod. It can share the pod's network and see its processes, so you get a shell with tools even when the app's image has none. with tools, into the running pod, sharing its network and (with --target) its process list:

$ kubectl debug hello-779c6b7b5-9qftf --image=busybox:1.37 --target=hello --container=dbg -- sh -c 'ps; wget -qO- localhost:8080/readyz; echo'
Targeting container "hello". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
PID   USER     TIME  COMMAND
    1 65532     0:00 /hello                          ← our app (user 65532, from the Dockerfile)
   24 root      0:00 sh -c ps; wget -qO- localhost:8080/readyz; echo
   34 root      0:00 ps
ready                                                ← localhost = the app's own network

For an interactive shell instead, use kubectl debug -it <pod> --image=busybox:1.37 --target=hello -- sh. You can see the app's processes and call it on localhost, without changing its image or restarting it.

Step 5: events — the whole namespace

When you don't know which object is unhappy, list recent events across the namespace, oldest first:

$ kubectl get events --sort-by=.lastTimestamp

Look for Warning lines: FailedScheduling, BackOff, Unhealthy, FailedMount, Evicted. One deployment's pods failing to schedule can be the real cause behind another service's errors.

Practice: the pod doctor

Pick a broken pod and walk through the diagnosis. All outputs are the real ones from our cluster:

$ kubectl get pods
NAME      READY   STATUS             RESTARTS      AGE
crashy    0/1     Error              3 (60s ago)   76s

Staff-level: Kubernetes in production

1. Shutting down without dropping requests

Every rollout, scale-down and node drain deletes pods. Here's what happens when a pod is deleted, and the race hidden in it:

When you delete a pod, the API server marks it as terminating, and two independent things start at once:

• Routing catches up. The endpoints controller marks the pod as terminating and not ready in the Service's EndpointSlice, then every node's kube-proxy and every ingress controller has to see that change and update its rules. That takes from milliseconds to a few seconds.
• The kubelet stops the container. It runs the preStop hook if there is one, then sends SIGTERM (the "please shut down" signalSignal (Unix)A tiny message the OS delivers to a running process, e.g. SIGINT (Ctrl+C), SIGTERM (“please shut down” — what kill and Kubernetes send first), SIGKILL (stop immediately; can't be caught) or SIGQUIT (Ctrl+\, Go prints every goroutine's stack). Send one with kill -TERM <pid>.), waits up to terminationGracePeriodSecondsterminationGracePeriodSecondsHow long Kubernetes waits after starting to stop a pod (preStop hook plus SIGTERM) before it sends SIGKILL. Default 30 seconds. (default 30 s) in total, then SIGKILL.

If SIGTERM arrives before routing has caught up, your app stops accepting connections while some nodes are still sending it new ones. The fix is a preStop hookpreStop hookA command or HTTP call Kubernetes runs inside a pod just before it sends SIGTERM — often a short sleep that gives load balancers time to stop sending traffic. that just waits a few seconds:

    spec:
      terminationGracePeriodSeconds: 30   # total budget: preStop + shutdown
      containers:
        - name: hello
          image: hello:v2
          lifecycle:
            preStop:
              sleep:
                seconds: 5                # keep serving while routing catches up

We measured it. With that hook, kubectl delete pod took 5.9 s, and the app's log shows SIGTERM arriving 5 seconds after the delete was issued at 23:05:24.98:

2026-10-30T23:05:22.150Z  ready
2026-10-30T23:05:30.053Z  SIGTERM: draining     ← preStop slept 5 s first
2026-10-30T23:05:30.055Z  bye

(Timestamps trimmed to milliseconds.) The built-in sleep action needs Kubernetes 1.30 or newer. On older clusters, people used exec: {command: ["sleep", "5"]}, which needs a sleep binary in the image, and a FROM scratch image like ours has none.

Then the app does its part on SIGTERM: stop accepting, finish in-flight requests, exit. Containers §09 covers the signal handling, and Worker pools §09 the draining. Our Go app does this (shown here with a 20 s drain budget):

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
defer stop()
<-ctx.Done() // ← blocks until SIGTERM arrives
log.Print("SIGTERM: draining")
shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
// srv is the *http.Server; earlier, main started it in its own goroutine with go srv.ListenAndServe()
if err := srv.Shutdown(shutdownCtx); err != nil { // stop accepting; wait for in-flight requests
	log.Printf("shutdown: %v", err) // the 20 s ran out with requests still open
}
log.Print("bye")

Keep the total under the grace period: 5 s preStop + up to 20 s drain < 30 s. If you need longer, raise terminationGracePeriodSeconds.

2. Voluntary disruptions: drains and PodDisruptionBudgets

Nodes get replaced all the time: OS patches, Kubernetes upgrades, the cluster autoscaler removing an empty node. Each starts with a cordon and drainCordon and drainPreparing a node for maintenance. Cordon marks it so no new pods are scheduled there; drain also evicts its pods so their controllers recreate them on other nodes.: no new pods there, and evictEvictionKubernetes removing a pod from a node on purpose — because the node is being drained for maintenance, or because the node is running out of memory or disk. The pod's controller then creates a replacement elsewhere. the existing ones so they're recreated elsewhere. If all three hello pods happen to be on that node, a drain could evict all three at once.

A PodDisruptionBudgetPodDisruptionBudget (PDB)A rule like “always keep at least 2 hello pods running” that voluntary disruptions — draining a node for an upgrade — must respect. It doesn't protect against crashes or a node dying. forbids that:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hello
spec:
  minAvailable: 2          # voluntary evictions must leave at least 2 ready pods
  selector:
    matchLabels:
      app: hello

We drained our node with the PDB in place (--pod-selector limited the drain to our app). Real output, trimmed:

$ kubectl get pdb
NAME    MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
hello   2               N/A               1                     22s
$ kubectl drain atlas-k8s-control-plane --pod-selector=app=hello --ignore-daemonsets --timeout=15s
node/atlas-k8s-control-plane cordoned
evicting pod default/hello-779c6b7b5-6cnlg
evicting pod default/hello-779c6b7b5-5w49p
evicting pod default/hello-779c6b7b5-9qftf
error when evicting pods/"hello-779c6b7b5-5w49p" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"hello-779c6b7b5-6cnlg" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/hello-779c6b7b5-9qftf evicted
…

One pod was allowed to go; the others were refused until a replacement became ready somewhere. (On our one-node cluster there was nowhere else to go, so the drain timed out and we uncordoned the node.) A PDB only covers voluntary disruptions. It doesn't stop a node from crashing, which is what the next setting is for.

3. Don't put every copy in one basket

The scheduler only promises your pods fit somewhere. By default nothing stops all three from landing on the same node, or in the same Availability zoneAvailability zone (AZ)One of a few separate data centers inside a cloud region (e.g. us-east-1a, 1b, 1c), with its own power and network, so one can fail without taking down the others., which a single failure then takes out. Topology spread constraintsTopology spread constraintsPod settings that ask the scheduler to spread a Deployment's pods evenly across nodes or availability zones, so one machine or zone failing can't take out every copy. ask for an even spread:

    spec:
      topologySpreadConstraints:
        - maxSkew: 1                                 # zones may differ by at most 1 pod
          topologyKey: topology.kubernetes.io/zone   # spread across zones (use kubernetes.io/hostname for nodes)
          whenUnsatisfiable: ScheduleAnyway          # prefer, don't refuse (DoNotSchedule = strict)
          labelSelector:
            matchLabels:
              app: hello

4. Least privilege by default

A container that's broken into should find as little as possible. These securityContextsecurityContextPod and container settings that limit what a process may do: run as a non-root user, forbid gaining privileges, make the filesystem read-only, drop Linux capabilities. settings cost nothing for most apps (our scratch image ran fine with all of them):

    spec:
      securityContext:
        runAsNonRoot: true                 # refuse to start if the image would run as root
      containers:
        - name: hello
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true   # the app can't modify its own files
            capabilities:
              drop: ["ALL"]                # no special Linux powers

Every pod runs as a ServiceAccountService account (Kubernetes)An identity for a program rather than a person. Kubernetes gives each pod a signed, short-lived token for its service account, which it can use to prove who it is. (its identity when calling the Kubernetes API), and gets that account's token mounted as a file. Add automountServiceAccountToken: false on pods that never call the Kubernetes API, so a compromised container can't use the pod's API credentials. Clusters can enforce a baseline with Pod Security Admission (namespace labels like pod-security.kubernetes.io/enforce: restricted). The image side of security is in Containers §11.

5. Not everything is a Deployment

Deployments are for stateless apps: any pod can be replaced by any other, and nothing is stored on its disk. Other shapes have their own controllers:

And you can add your own. An operatorOperator (and CRD)A custom controller that teaches Kubernetes to manage something new, such as a Postgres cluster. It adds a new object type (a CustomResourceDefinition, CRD) and reconciles it like the built-in ones. adds a new object type (a CRD) plus a controller that reconciles it, so you can write kind: PostgresCluster and let the operator handle replicas, backups and failover. That's the same reconcile loop from §05, applied to anything.

6. Running the cluster itself

• Use managed KubernetesManaged Kubernetes (EKS, GKE, AKS)A cloud provider runs and upgrades the control plane for you (Amazon EKS, Google GKE, Azure AKS); you pay for and manage the worker nodes (or not, in their serverless modes). unless you have a platform team: the provider runs etcd, the API server and control-plane upgrades.
• Upgrade regularly. Kubernetes releases a minor version about three times a year, and each gets roughly 14 months of patches upstream. Falling several versions behind turns upgrades into projects. Clusters upgrade one minor version at a time.
• Keep everything in git. Manifests, Helm values, cluster add-ons. Rebuilding a cluster from git is your disaster recovery, and GitOpsGitOpsRunning deployments by committing the desired configuration (Kubernetes manifests) to a git repo and letting a tool apply whatever is in git to the cluster. tools (Argo CD, Flux) make "what's in git" and "what's running" the same thing, continuously reconciled.
• Watch cost. Requests reserve capacity whether you use it or not. Inflated requests (§10) mean paying for idle nodes.

Check yourself

Predict the answer before you click. Every one is covered above: Q1 in §05, Q2 in §06, Q3 in §09 and §11, Q4 in §10, Q5 in §08, Q6 in §11, Q7 in §14.

strategy:
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

Cheat sheet