EKS Node Groups

The control plane is AWS's problem. The nodes are yours, and how you run them determines your cost, your upgrade pain and how quickly pods get scheduled.

Three ways to run workers

Managed node groups are the default choice. AWS handles the Auto Scaling group, the AMI, and a graceful drain-and-replace upgrade.

Self-managed is for what managed cannot express: a custom AMI with specific kernel settings, unusual GPU configurations, or Windows in older setups.

Fargate removes nodes entirely, and its limitations are what decide it. No DaemonSets, so any log shipper, security agent or service mesh sidecar delivered that way does not run. No privileged containers, no host networking, and a maximum of 4 vCPU and 30 GB per pod. It suits bursty, isolated workloads; it rarely suits a general-purpose cluster.

A managed node group

eksctl create nodegroup --config-file=- <<'EOF'
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: prod
  region: eu-west-1
managedNodeGroups:
  - name: general-v2
    instanceTypes: ["m6i.large", "m6a.large", "m5.large"]
    minSize: 2
    maxSize: 20
    desiredCapacity: 3
    privateNetworking: true
    volumeSize: 100
    volumeType: gp3
    amiFamily: AmazonLinux2023
    labels:
      workload: general
    updateConfig:
      maxUnavailablePercentage: 25
    iam:
      withAddonPolicies:
        cloudWatch: true
EOF

List several instance types. It widens the pool the group can draw from, which matters for capacity during a busy period and enormously for Spot.

Include a version in the name. Node group configuration is largely immutable. Instance types, AMI family and subnets cannot be changed after creation, so "upgrading" often means creating general-v2, migrating, and deleting general. A name without a version makes that awkward.

Taints and labels

Labels attract pods; taints repel them. Together they keep workloads where they belong:

managedNodeGroups:
  - name: gpu
    instanceTypes: ["g5.xlarge"]
    labels:
      workload: ml
      nvidia.com/gpu: "true"
    taints:
      - key: nvidia.com/gpu
        value: "true"
        effect: NoSchedule

The taint is what stops ordinary pods landing on expensive GPU instances. Without it the scheduler treats them as general capacity and you pay GPU rates to run a web service. A mistake that shows up on the bill rather than in any alert.

# In the workload
nodeSelector:
  workload: ml
tolerations:
  - key: nvidia.com/gpu
    operator: Equal
    value: "true"
    effect: NoSchedule

Karpenter over Cluster Autoscaler

For node scaling there are two options, and they are not close any more.

Cluster Autoscaler adjusts the desired capacity of existing node groups. It can only launch instance types those groups already define, needs a group per instance shape, and takes several minutes.

Karpenter watches unschedulable pods and launches instances chosen to fit them, right-sizing automatically, consolidating underused nodes, and provisioning in under a minute.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s

Rather than enumerating instance types, you describe acceptable characteristics and let Karpenter pick. consolidationPolicy then actively repacks pods onto fewer nodes and terminates the remainder, which is where most of the saving comes from. Cluster Autoscaler only scales down nodes that are already nearly empty.

Keep a small managed node group for Karpenter itself and CoreDNS. Karpenter cannot provision the nodes it runs on.

Spot instances

Spot is where cluster cost drops by 60 to 90%, and it is safe for stateless workloads given three things.

Diversify. Accept many instance types and families so an interruption in one pool is absorbed elsewhere. A Spot node group pinned to a single type is what gives Spot its reputation.

Handle interruptions. AWS gives a two-minute warning; something must cordon and drain the node. Karpenter handles this natively via an SQS queue; with managed node groups, run the AWS Node Termination Handler.

Set PodDisruptionBudgets, or a drain can evict every replica at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

Mix capacity types rather than going all-Spot: a baseline of on-demand for the workloads that must not move, Spot for everything else.

Upgrading nodes

aws eks update-nodegroup-version --cluster-name prod \
  --nodegroup-name general --kubernetes-version 1.32

AWS cordons each node, drains it respecting PodDisruptionBudgets, waits, then terminates it.

That respect for PDBs is a double edge. A PDB that can never be satisfied, minAvailable: 1 on a single-replica Deployment, blocks the drain indefinitely, and the upgrade hangs for hours before failing:

Before any upgrade, check for PDBs that cannot be met:

kubectl get pdb -A

ALLOWED DISRUPTIONS: 0 is the row that will stop your upgrade. --force skips PDBs entirely, which gets the upgrade moving and causes the outage the PDB existed to prevent.

Set maxUnavailablePercentage to control the pace: 25% replaces a quarter of the group at a time, which is much faster than the default of one node while still being gradual.

Diagnosing a node that will not join

A node that launches but never appears in kubectl get nodes is nearly always one of four things:

kubectl get nodes
aws eks describe-nodegroup --cluster-name prod --nodegroup-name general \
  --query 'nodegroup.health.issues'

• IAM: the node role is missing AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly or AmazonEKS_CNI_Policy.
• Networking: a private subnet with no NAT gateway and no VPC endpoints cannot reach the EKS API or ECR.
• Security groups: the cluster and node groups must permit traffic in both directions.
• User data: a custom launch template whose user data does not call the bootstrap script.

describe-nodegroup's health.issues usually names it directly, and is faster than reading instance logs.

For a node that joined and later went NotReady, look at the kubelet:

kubectl describe node ip-10-0-1-42.ec2.internal | grep -A10 Conditions

DiskPressure is common and usually means image accumulation. Set imageGCHighThresholdPercent or use larger root volumes.

Common errors

IAM, networking, security groups or bootstrap user data.

An unsatisfiable PodDisruptionBudget. Fix the PDB or the replica count.

The pod lacks a toleration for a tainted node group.

Nodes at Ready but pods stay Pending. Often CNI IP exhaustion rather than a node problem.

Quick reference

Key takeaways

• Managed node groups are the default. Fargate's lack of DaemonSets rules it out for most clusters.
• List several instance types: essential for Spot, useful for capacity generally.
• Taint expensive node groups, or ordinary pods will schedule onto GPU instances.
• Prefer Karpenter to Cluster Autoscaler: faster, right-sizing, and it consolidates.
• Spot needs diversification, interruption handling and PodDisruptionBudgets.
• Check kubectl get pdb -A before upgrading: ALLOWED DISRUPTIONS: 0 blocks the drain.
• --force skips PDBs and causes the outage they were preventing.
• describe-nodegroup --query 'nodegroup.health.issues' names most join failures directly.

Check your understanding

<MCQExercise exercise={{ id: 'aws-eks-ng-quiz1', title: 'An upgrade that hangs for hours', description: 'A managed node group upgrade stalls, then fails with "pods that cannot be evicted". One namespace has a single-replica Deployment with a PodDisruptionBudget of minAvailable: 1. What is happening?', options: [ { id: 'a', text: 'The PDB can never be satisfied with one replica, so the drain cannot evict that pod and the upgrade blocks until it times out' }, { id: 'b', text: 'Managed node groups do not support PodDisruptionBudgets and the PDB must be deleted before any upgrade' }, { id: 'c', text: 'The node group needs maxUnavailable set to 100% for upgrades to proceed' }, { id: 'd', text: 'Single-replica Deployments must be converted to StatefulSets before a node upgrade' }, ], correctId: 'a', explanation: 'EKS drains each node respecting PodDisruptionBudgets, which is what makes managed upgrades safe. A budget requiring one pod to stay available on a Deployment that only has one pod means evicting it would violate the budget, so the eviction is refused indefinitely and the upgrade waits. Run kubectl get pdb -A before upgrading and look for ALLOWED DISRUPTIONS: 0. The right fix is scaling to two replicas or relaxing the budget, not --force, which skips PDBs entirely and causes exactly the outage the budget existed to prevent.', wrongExplanations: { b: 'Managed node groups honour PDBs deliberately. Deleting them removes the protection rather than fixing the configuration.', c: 'maxUnavailable controls how many nodes are replaced at once and does not override an eviction refusal.', d: 'Workload kind is irrelevant. The conflict is between replica count and the budget.', }, }} />

<MCQExercise exercise={{ id: 'aws-eks-ng-quiz2', title: 'Web pods on GPU nodes', description: 'A g5.xlarge node group was added for machine learning. Ordinary web pods are now scheduling onto it and the bill has risen sharply. The node group has the label workload: ml. What is missing?', options: [ { id: 'a', text: 'A taint on the node group. Labels attract pods that select them but do nothing to repel pods that do not' }, { id: 'b', text: 'A nodeSelector on the web pods excluding the ml label' }, { id: 'c', text: 'Resource limits on the web pods, without which the scheduler prefers the largest nodes' }, { id: 'd', text: 'A separate cluster, since GPU and general workloads cannot share one' }, ], correctId: 'a', explanation: 'Labels and taints do opposite jobs. A label lets a pod ask for that node with a nodeSelector or affinity rule, but it places no restriction on pods that say nothing, to the scheduler an untainted GPU node is simply available capacity. Add a taint such as nvidia.com/gpu=true:NoSchedule so only pods carrying a matching toleration can land there, and give the ML workloads both the toleration and the nodeSelector. This is a cost problem that shows up on the invoice rather than in any alert, which is why it often runs for weeks.', wrongExplanations: { b: 'Adding exclusions to every other workload in the cluster does not scale and breaks the moment someone forgets.', c: 'The scheduler does not prefer larger nodes; it places pods wherever requests fit.', d: 'Mixed node groups in one cluster are a normal and supported pattern when taints are used correctly.', }, }} />