Understand Kubernetes architecture: API server, etcd, scheduler, controller manager, kubelet and kube-proxy, and what happens when you run kubectl apply.
A Kubernetes cluster splits into two halves. The control plane decides what should happen. The nodes make it happen.
Knowing which component does what is not trivia. It is what turns "the cluster is broken" into a specific thing to check.
Entry point for all requests
Assigns pods to nodes
Reconciles desired state
Cloud Controller Manager
Integrates with cloud providers (AWS, GCP, Azure)
The only door into the cluster. Every component, kubectl, controllers, the scheduler, the kubelets on every node, talks exclusively to the API server, and none of them talk to each other or to etcd directly.
It authenticates the caller, authorises the request against RBAC, runs admission controllers that can mutate or reject the object, validates it, and writes it to etcd.
It is stateless, so it scales horizontally, and production clusters run several behind a load balancer.
When it fails: you lose all control. kubectl stops working, no deployments, no scaling, no self-healing. Running Pods keep running and keep serving traffic, because nodes do not need it to keep doing what they are already doing.
A distributed key-value store holding the entire cluster state. Every object, its spec and its status. Nothing else in Kubernetes is stateful. Lose etcd without a backup and you have lost the cluster, even with every node still healthy.
It uses Raft consensus, which is why you run an odd number of members. A three-member cluster tolerates one failure; five tolerates two. Four is worse than three. More nodes to fail, the same tolerance.
etcd is sensitive to disk latency in a way people underestimate. Slow disks produce leader elections, which look like random cluster-wide instability. Give it fast SSDs.
On a managed cluster, EKS, GKE, AKS, the provider runs and backs up etcd, which is a large part of what you are paying for.
Watches for Pods with no node assigned and picks one. It does not start anything; it writes the chosen node onto the Pod, and that node's kubelet does the rest.
Two phases: filtering removes nodes that cannot run the Pod (not enough free CPU or memory, a taint with no matching toleration, an unsatisfiable node selector or affinity rule, no access to the required volume), then scoring ranks the survivors and the highest wins.
When it fails: new Pods sit in Pending forever. Everything already running is unaffected.
That is also the clue when a single Pod is stuck Pending. The scheduler filtered out every node, and kubectl describe pod reports exactly why:
One binary running many controllers, each an independent loop: observe current state, compare to desired state, act on the difference.
When it fails: nothing self-heals. Deleted Pods are not replaced, failed nodes are not drained, deployments do not progress. Existing workloads keep serving.
Where present, this is the part that talks to the cloud provider: provisioning load balancers for LoadBalancer Services, attaching storage volumes, and updating node metadata such as region and zone.
When it fails: LoadBalancer Services stay <pending> and volumes do not attach.
# Back it up. This is the one thing you cannot recreate.
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.keyThe agent on every node. It watches the API server for Pods assigned to it, tells the container runtime to start them, mounts their volumes, runs their probes, and reports status back.
It is the component that actually runs your containers, and it is deliberately independent: a kubelet that cannot reach the API server keeps its existing containers running. That is why a control plane outage does not stop traffic.
When it fails: the node stops reporting and is marked NotReady. After roughly five minutes the node controller starts evicting its Pods so they can be rescheduled elsewhere, but the containers on the failed node may still be running, which is why a partitioned node is a genuinely awkward failure.
Implements Services. It watches Services and EndpointSlices and programs the node's iptables or IPVS rules so traffic to a Service's virtual IP is rewritten to a real Pod IP.
The key insight: there is no proxy process in the request path. kube-proxy writes kernel rules and gets out of the way. That is why Service load balancing is per-connection and cannot inspect HTTP.
When it fails: existing rules stay in place so current Services keep working, but new Services and changed endpoints are not programmed.
Actually runs the containers, containerd most commonly, or CRI-O. Kubernetes talks to it through the Container Runtime Interface.
Docker was removed as a supported runtime in Kubernetes 1.24. This changed nothing about images: Docker builds OCI images and containerd runs them.
kubectl get nodes
kubectl describe node <name> # conditions: MemoryPressure, DiskPressure, ReadyRequest flow: kubectl → running pod
Validates & persists to etcd
Pulls image & starts container
✓ Desired state achieved
Validates & persists to etcd
Pulls image & starts container
✓ Desired state achieved
Nothing in that chain is a direct call between components. Each one watches the API server and reacts. That is why Kubernetes is resilient to any single piece being briefly unavailable, and why "eventually consistent" describes its behaviour better than "immediate".
Control plane nodes normally carry a taint preventing workloads from being scheduled there. Removing it on a single-node dev cluster is standard; doing it in production means an application can starve the control plane.
kube-scheduler crashes and does not restart. What is the observable effect on the cluster?
A team runs a four-member etcd cluster for extra safety. How many member failures can it tolerate while still accepting writes?
Compare ways to install Kubernetes: kind, Minikube, Docker Desktop, k3s, kubeadm and managed clusters, and pick the right one for learning or production.
Install Minikube, run minikube start, enable add-ons like ingress, reach services with minikube tunnel, and build images locally without a registry.
Install kind and run kind create cluster to get a local Kubernetes cluster in Docker. Covers multi-node configs, pinned versions, local images, ingress and CI.