Kubernetes node NotReady
Fix a NotReady node: run kubectl describe node, then check disk and memory pressure, the kubelet, container runtime and CNI. Includes journalctl commands.
$ kubectl get nodes
Incident Overview
KubernetesKubernetes
Fix a NotReady node: run kubectl describe node, then check disk and memory pressure, the kubelet, container runtime and CNI. Includes journalctl commands.
medium fix·8 min read·5 causes
By Vishvesh Patel·DevOps Engineer·Updated 16 September 2026
$ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-1-42 NotReady <none> 14d v1.31.3 ip-10-0-2-17 Ready <none> 14d v1.31.3 Conditions: Type Status Reason Message Ready False KubeletNotReady container runtime network not ready DiskPressure True KubeletHasDiskPressure
Do this first3 steps
Run these in order. Each one tells you what its output means before you change anything.
• Read the node conditions
DiskPressure, MemoryPressure or PIDPressure names the exhausted resource directly. Every condition False with Ready Unknown means the kubelet has stopped reporting. The node or its network is gone.
• If it is disk, check inodes as well as bytes
Images exhaust inodes before they exhaust space, and df -h alone will look healthy. sudo crictl rmi --prune reclaims unused images.
• If nothing is under pressure, go to the kubelet
A dead containerd and an expired client certificate both surface here. Check the certificate dates with sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates.
kubectl describe node ip-10-0-1-42 | grep -A8 Conditions
Then: the 5 causes in full
If the steps above did not settle it, these are the causes in the order they are usually responsible. Each section explains why it happens and how to confirm it.
NotReady means the control plane is no longer satisfied that the node can run workloads. After roughly five minutes it taints the node node.kubernetes.io/unreachable and begins evicting pods, so this becomes a capacity problem quickly.
The first question is always whether the node is unhealthy or merely unable to talk to the API server. Those look identical from kubectl and have completely different causes.
Read the conditions
Unknown with NodeStatusUnknown means the kubelet has stopped reporting at all. The node may be perfectly healthy and simply unreachable. Check LastHeartbeatTime against now to see when it went quiet.
False with a specific reason means the kubelet is reporting and telling you what is wrong. That is much better, because the reason names the cause:
kubectl describe node ip-10-0-1-42 | grep -A8 Conditions
Cause 1: Disk pressure
The most common cause on a long-lived node, and the one that resolves itself least often.
The kubelet starts evicting at 85% by default and stops scheduling well before that. Two things fill a node: container images accumulating over months, and container logs.
The durable fixes are larger root volumes, log rotation configured in the container runtime, and kubelet image garbage collection thresholds set deliberately rather than left at defaults. Remember that inodes can exhaust while space remains, which df -i shows.
kubectl describe node ip-10-0-1-42 | grep -i pressure
Cause 2: Memory pressure
Limits far above requests is the setup for this. Pods are scheduled against requests, so a node can accept far more work than it can actually run when every pod bursts toward its limit, and the kubelet then evicts to protect itself.
Check whether the OOM killer has been active:
The fix is honest requests, and --kube-reserved plus --system-reserved on the kubelet so the node keeps memory for itself. Pods without requests at all are the worst case, since the scheduler treats them as needing nothing.
kubectl describe node ip-10-0-1-42 | grep -A5 "Allocated resources"
Cause 3: The kubelet is not running
Look for the reason it stopped. Frequent causes:
sudo systemctl status kubelet sudo journalctl -u kubelet --no-pager -n 100
Cause 4: The container runtime is down
The kubelet cannot report ready without a working runtime. containerd failing usually shows in its own journal, and a full disk is a common underlying cause, which is why disk pressure and runtime failure often appear together.
Ready False KubeletNotReady container runtime is down
Cause 5: CNI not ready
The CNI DaemonSet pod on that node has failed or was never scheduled. On EKS that is aws-node; check its logs for IAM or IP exhaustion problems. An empty /etc/cni/net.d/ means the plugin never wrote its configuration.
This is the normal state for a brand new node for the first minute or so. It is only a problem if it persists.
Ready False KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Is it just unreachable?
If conditions are Unknown, the node may be fine:
Causes: a security group or firewall change, a NAT gateway failure, a route table change, or the API server endpoint being unreachable from that subnet. In cloud environments a node that is Unknown while the instance is healthy is very often a networking change rather than anything on the node.
# From a machine that can reach the node ssh ip-10-0-1-42 'sudo systemctl is-active kubelet && uptime'
Recovering
Drain it first, so pods move rather than being evicted abruptly:
If the drain hangs, a PodDisruptionBudget is blocking it. Check kubectl get pdb -A for one showing ALLOWED DISRUPTIONS: 0.
Then fix or replace:
Replacing is usually the right call. A node that filled its disk once will do so again, and an Auto Scaling group brings up a clean one in minutes. Investigate the cause on the replacement, not on the casualty.
kubectl drain ip-10-0-1-42 --ignore-daemonsets --delete-emptydir-data --timeout=300s
Preventing it
Alarm on node conditions, not just node count. kube_node_status_condition in kube-state-metrics gives you DiskPressure and MemoryPressure before NotReady.
Set requests honestly, and set --kube-reserved and --system-reserved.
Rotate container logs and configure image garbage collection.
Replace nodes regularly. A fleet where no node is older than a few weeks does not accumulate the conditions that cause this, which is one of the better arguments for Karpenter's consolidation or a scheduled instance refresh.
A checklist
Frequently Asked Questions
What does node NotReady mean in Kubernetes?
The control plane no longer believes the node can run workloads, either because the kubelet reported a problem or because it stopped reporting at all. After about five minutes the node is tainted node.kubernetes.io/unreachable and its pods begin being evicted, so capacity is affected. kubectl describe node shows the conditions, and whether they read False with a reason or Unknown tells you immediately which of those two situations you are in.
What is the difference between a node condition of False and Unknown?
False with a specific reason means the kubelet is alive and actively reporting a problem, disk pressure, memory pressure, a runtime or CNI failure, which is comparatively good news because the reason names the cause. Unknown with NodeStatusUnknown means no heartbeat has arrived, so the node may be entirely healthy and simply unable to reach the API server. Compare LastHeartbeatTime with the current time to see when it went quiet.
Why is my node showing DiskPressure?
The kubelet's root or image filesystem has crossed its eviction threshold, 85% by default. On a long-lived node the two things that fill it are accumulated container images and container logs. Check df -h /var/lib/containerd and also df -i, since inodes can exhaust while space remains. crictl rmi --prune reclaims unused images immediately; the durable fixes are log rotation, deliberate image garbage collection thresholds, and larger root volumes.
Why does a node show MemoryPressure when pods have limits set?
Because scheduling uses requests, not limits. A node can accept far more pods than it can actually run if limits are set well above requests, and when several pods burst toward their limits simultaneously the kubelet evicts to protect itself. Compare allocated requests against limits in kubectl describe node. Set requests close to real usage, configure --kube-reserved and --system-reserved, and treat pods with no requests at all as the worst offenders.
Should I fix a NotReady node or replace it?
Replace it, in most cases. A node that filled its disk or exhausted memory once will do so again, and an Auto Scaling group produces a clean replacement in minutes. Investigating on the replacement is safer than experimenting on a node that is currently carrying workloads. Drain first with kubectl drain --ignore-daemonsets so pods move gracefully, then delete the node object and terminate the instance. Repair in place is worth it mainly when the cause is a configuration change you can reverse.
Why is a brand new node NotReady with a CNI error?
Because the CNI plugin has not finished initialising. Until the network plugin writes its configuration to /etc/cni/net.d/ and reports ready, the kubelet cannot mark the node ready, so NetworkPluginNotReady is the normal state for the first minute or so after a node joins. It is only a problem if it persists, at which point check the CNI DaemonSet pod on that node, which on EKS means aws-node and often turns out to be an IAM permission or IP exhaustion issue.
Reference and practice
Learn the underlying concept
Other Kubernetes errors
Free, comprehensive DevOps tutorials and learning roadmaps. Master Docker, Kubernetes, CI/CD, and more.
Get notified about new tutorials and features.
• Roadmaps
• Interview Prep
• Certifications
• Company
© 2026 DevOpsLesson. All rights reserved.
Need to practice this scenario?
Simulate this error condition and practice recovery commands safely in our in-browser terminal.