Kubernetes Requests and Limits

Two numbers per container decide almost everything about how your workload behaves under pressure: requests, which drive scheduling, and limits, which cap consumption at runtime.

Get them wrong and you get nodes that look idle while Pods cannot schedule, containers killed at 2am, or latency that appears only under load and makes no sense from the application's own metrics.

Requests and limits are two different mechanisms

The scheduler only ever looks at requests. It sums the requests of everything already on a node and places your Pod where the remaining capacity fits. It never looks at actual usage.

That is why a node can show 20% real CPU usage in monitoring and still refuse to schedule anything. Its requests are fully committed. kubectl describe node shows both, and the gap between them is usually the story:

kubectl describe node ip-10-0-1-42 | grep -A8 "Allocated resources"

Limits exceeding 100% is normal, that is deliberate overcommit, betting that not everything peaks simultaneously. Requests at 95% means this node is effectively full.

Units

CPU is measured in cores, and m means millicores. Thousandths of a core.

CPU is compressible: exceed your limit and you are throttled, not killed.

Memory is measured in bytes with suffixes. The distinction between the two families catches people:

Use Mi and Gi. Memory is incompressible: exceed the limit and the kernel kills the process. There is no throttling option.

What actually happens at the limit

Exceeding a CPU limit throttles the container. The kernel simply stops scheduling it for the remainder of each 100ms period. Nothing crashes; requests just get slower. This is genuinely hard to diagnose from inside the application, because the process sees time passing with no work done.

kubectl exec -it <pod> -- cat /sys/fs/cgroup/cpu.stat | grep throttled

A high nr_throttled relative to nr_periods means the limit is costing you latency.

Exceeding a memory limit gets the container OOMKilled immediately.

kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

A Pod that restarts periodically with no error in its own logs is the signature. The application is killed without warning, so it never gets to log anything.

The CPU limit argument

Setting CPU requests is uncontroversial. Do it always.

Setting CPU limits is genuinely contested. The case against: throttling adds latency, and multi-threaded runtimes hit the limit sooner than intuition suggests because the quota is consumed across all threads. A service limited to 1000m running eight threads can exhaust its quota in 12ms of wall-clock time and then stall for 88ms.

The case for: without limits, one badly behaved workload starves its neighbours, and performance becomes unpredictable in a way that depends on what else happens to be scheduled nearby.

A defensible middle position: always set memory limits, an unbounded leak takes down the whole node, and set CPU requests but be cautious with CPU limits, particularly on latency-sensitive services. If you do set CPU limits, set them well above observed peak, not at it.

QoS classes

Kubernetes assigns each Pod a class from its requests and limits, and that class decides eviction order when a node runs out of memory.

kubectl get pod <name> -o jsonpath='{.status.qosClass}'

BestEffort Pods are evicted first and are also placed carelessly, since the scheduler thinks they need nothing. A production workload with no requests set is not merely unoptimised. It is first in line to be killed under pressure.

For anything critical, Guaranteed gives the strongest protection: set requests equal to limits.

Namespace-level controls

LimitRange supplies defaults and bounds within a namespace, so a Pod submitted with nothing set does not become BestEffort:

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
spec:
  limits:
    - type: Container
      default:              # applied as the limit if unset
        cpu: 500m
        memory: 512Mi
      defaultRequest:       # applied as the request if unset
        cpu: 100m
        memory: 128Mi
      max:
        cpu: "4"
        memory: 8Gi

ResourceQuota caps the namespace in aggregate, which is how you stop one team consuming a shared cluster:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"

One consequence worth knowing: once a ResourceQuota constrains a resource in a namespace, every Pod must specify it. A Pod with no memory request is rejected outright. Pair a ResourceQuota with a LimitRange so existing manifests keep working.

Choosing the numbers

Measure rather than guess:

kubectl top pods --containers
kubectl top nodes

A reasonable starting point:

• Memory request: observed steady-state usage plus roughly 25%.
• Memory limit: request plus headroom for real peaks. For JVM, Go or Node applications, make sure the runtime knows about the limit or it will size its heap against the node's total memory.
• CPU request: observed average, not peak.
• CPU limit: well above peak, or omitted for latency-sensitive services.

kubectl top needs metrics-server installed; without it the command fails. For real sizing you want historical data from Prometheus, since a single snapshot misses daily peaks.

Vertical Pod Autoscaler in recommendation mode is useful here: it observes actual usage and suggests values without changing anything.

Explore further in this section

Quick reference

Key takeaways

• The scheduler reads requests, never actual usage. A node at 20% CPU can still be unschedulable.
• Mi is not M. Use the binary suffixes.
• CPU throttles, memory kills. A silent periodic restart with clean logs is almost always OOMKilled.
• Always set memory limits. Be cautious with CPU limits on latency-sensitive services.
• No requests means BestEffort, which is scheduled carelessly and evicted first.
• Guaranteed, requests equal to limits, is the strongest protection for critical workloads.
• A ResourceQuota makes requests mandatory. Add a LimitRange alongside it or existing manifests break.

Check your understanding

<MCQExercise exercise={{ id: 'k8s-res-scheduler-requests', title: 'A full node that looks idle', description: 'Monitoring shows a node at 20% CPU utilisation, but new Pods will not schedule there and report Insufficient cpu. Why?', options: [ { id: 'a', text: 'The scheduler allocates based on the sum of CPU requests, not actual usage, and those requests are already committed' }, { id: 'b', text: 'The metrics are stale and the node is genuinely at capacity' }, { id: 'c', text: 'The node has a taint that prevents new Pods from being scheduled' }, { id: 'd', text: 'CPU limits on existing Pods are reserving capacity that cannot be reused' }, ], correctId: 'a', explanation: 'Scheduling is an accounting exercise over requests. If Pods on a node request 3800m in total, that capacity is committed regardless of whether they use it, and the scheduler will not place anything that does not fit in the remainder. Requests that are far above real usage are the most common cause of expensive, half-idle clusters.', wrongExplanations: { b: 'The message is about allocatable capacity rather than measured load, so both figures can be accurate at the same time. They are measuring different things.', c: 'A taint produces a distinct message naming the taint. "Insufficient cpu" specifically means the requests did not fit.', d: 'Limits are not reserved and may sum well above 100% of a node. Only requests participate in scheduling decisions.', }, }} />

<MCQExercise exercise={{ id: 'k8s-res-qos-eviction', title: 'Which Pod gets evicted', description: 'A node is under memory pressure. Three Pods are running: A with no resources set, B with requests below its limits, and C with requests equal to its limits. Which is evicted first?', options: [ { id: 'a', text: 'A, because with nothing set it is BestEffort, the lowest QoS class' }, { id: 'b', text: 'C, because it has the largest reservation and freeing it recovers the most memory' }, { id: 'c', text: 'B, because it is exceeding its requests and is therefore over its fair share' }, { id: 'd', text: 'Whichever Pod started most recently, regardless of QoS class' }, ], correctId: 'a', explanation: 'QoS class determines eviction order: BestEffort first, then Burstable, then Guaranteed. A Pod with no requests or limits told Kubernetes it needs nothing, so it is both placed carelessly and sacrificed first. This is why production workloads should always set requests, independent of any performance tuning.', wrongExplanations: { b: 'Guaranteed is the most protected class, not the least. Setting requests equal to limits is exactly how you make a Pod an eviction candidate of last resort.', c: 'Exceeding requests is normal and expected for Burstable Pods, that is what bursting means. Burstable is evicted after BestEffort, not before.', d: 'Start time is only used as a tie-break within the same QoS class and usage profile. Class dominates the decision.', }, }} />