Evicted: The node was low on resource: ephemeral-storage

KubernetesKubernetes

Fix Pod eviction for ephemeral-storage: set requests and limits, cap emptyDir with sizeLimit, rotate container logs and tune image garbage collection.

medium fix·6 min read

By Vishvesh Patel·DevOps Engineer·Updated 21 September 2026

Run these in order. Each one tells you what its output means before you change anything.

• Find out whether it was the Pod's limit or the node's threshold

"exceeds the total limit of containers" means your own ephemeral-storage limit was breached. "The node was low on resource" means the node hit DiskPressure and the kubelet chose a victim, which may not be the Pod at fault.

• Check disk on the node itself

A node reporting DiskPressure true is evicting Pods to reclaim space. Fixing one Pod will not help if the node is full for another reason, such as accumulated image layers.

• Clear the usual culprit, unused images, on the node

Image layers under /var/lib/containerd or /var/lib/docker are the most common consumer. The kubelet garbage-collects them at its own thresholds, which are often reached later than you would like.

Ephemeral storage is the node's local disk used by Pods: container writable layers, emptyDir volumes, and logs written to stdout. It is not a PersistentVolume. When it runs short, the kubelet evicts Pods to get it back.

Two distinct mechanisms produce an eviction, and they need different fixes.

You set limits.ephemeral-storage and the Pod exceeded it. This is your Pod's fault and only your Pod is affected.

The node's threshold.

The node hit DiskPressure and the kubelet picked victims. Note the phrase exceeds its request of 0: with no requests.ephemeral-storage set, any usage exceeds the request, which makes the Pod a preferred eviction candidate. Pods that declare a request and stay under it are evicted last.

On the node, four things consume it:

kubectl debug node mounts the host filesystem under /host, which is the supported way in without SSH.

In practice it is nearly always image layers on a long-lived node, or one Pod writing an enormous log.

The request protects you: the scheduler reserves it, and eviction ranking puts Pods under their request last. The limit protects the node from you.

The tradeoff is real. A Pod that exceeds its ephemeral-storage limit is evicted immediately with no grace, so set the limit above your genuine peak. A batch job that writes a large temporary file needs headroom for it.

An emptyDir with no sizeLimit can consume the whole node.

For caches and scratch space that fit comfortably in RAM, a memory-backed emptyDir avoids disk entirely:

Be careful: memory-backed emptyDir counts against the container's memory limit, so writing 512Mi into it with a 256Mi memory limit gets you OOMKilled rather than a disk eviction. Raise the memory limit accordingly.

Everything a container writes to stdout is stored on the node. A debug-level log in production fills a disk quickly, and the default rotation is more generous than people expect.

The kubelet rotates container logs with containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5), so 50Mi per container. With 40 Pods on a node that is 2Gi before anything else.

The application-side fix is better: log at info in production, and write large diagnostic output to an object store rather than stdout.

The kubelet removes unused images only when disk usage crosses a threshold:

At 85% full it deletes unused images until it reaches 80%. On a node that accumulates many image versions, lowering these reclaims space earlier:

Note that the eviction threshold defaults to nodefs.available<10%, which is above the image GC high threshold in disk-full terms. So a node can start evicting Pods while image GC has already run and found nothing more to delete. If that is your pattern, the node's disk is genuinely too small.

Evicted Pods stay in the API as a record. They consume no resources but clutter everything:

The kubelet's --pod-eviction-timeout and the controller's terminated-pod GC handle this eventually, at a threshold of 12,500 Pods by default, which is why they appear to accumulate forever on a small cluster.

What counts as ephemeral storage in Kubernetes?

The node's local disk used by Pods: each container's writable layer, emptyDir volumes, and anything written to stdout or stderr, which the kubelet stores as files under /var/log/pods. It explicitly does not include PersistentVolumes, which are separate and not subject to eviction. Image layers live on the same filesystem and count towards the node filling up, though they are not attributed to any individual Pod, which is why a node can hit DiskPressure when no single Pod is using much.

Why was my Pod evicted when a different Pod filled the disk?

Because node-level eviction is about reclaiming space, not about blame. When the kubelet hits its eviction threshold it ranks Pods by how far their usage exceeds their request and by QoS class, then evicts from the top. A Pod with no requests.ephemeral-storage exceeds its request of zero by definition, so it ranks as a candidate regardless of whether it used much. Setting a realistic request is the single most effective way to avoid being chosen as the victim.

How do I limit how much disk a Pod can use?

Set limits.ephemeral-storage in the container's resources, and sizeLimit on any emptyDir volume. The limit is enforced strictly: exceeding it evicts the Pod immediately with no grace period, so set it above your real peak rather than at it. Also set requests.ephemeral-storage, which both reserves capacity at scheduling time and improves your position in the eviction ranking when the node comes under pressure for reasons that have nothing to do with you.

Does a memory-backed emptyDir avoid the disk problem?

It avoids the disk, and moves the cost to memory. emptyDir with medium: Memory is a tmpfs, so it never touches the node's disk, but its contents count against the container's memory limit. Writing 512Mi into one with a 256Mi memory limit gets the container OOMKilled instead of evicted, which is a worse failure because it is less obviously related to the volume. It is a good option for small caches and scratch space, provided you raise the memory limit to cover the sizeLimit.

Why does the node still run out of disk after image garbage collection?

Because the kubelet's image GC high threshold, 85% by default, is less aggressive than the eviction threshold, nodefs.available<10%. Image GC only removes images not used by any container, so a node running many distinct images has little it can reclaim. If GC has run and the node still evicts, the honest conclusion is usually that the node's disk is too small for the workload density. Lowering imageGCHighThresholdPercent helps nodes that accumulate many versions of the same image over time.

Free, comprehensive DevOps tutorials and learning roadmaps. Master Docker, Kubernetes, CI/CD, and more.

Get notified about new tutorials and features.

• Roadmaps

• Interview Prep

• Certifications

• Company

© 2026 DevOpsLesson. All rights reserved.