EKS nodes not joining the cluster
Fix EKS nodes not joining: attach the node IAM policies, open the API server path, check security groups and subnet tags, then read the instance bootstrap logs.
NodeCreationFailure: Instances failed to join the kubernetes cluster
Incident Overview
Fix EKS nodes not joining: attach the node IAM policies, open the API server path, check security groups and subnet tags, then read the instance bootstrap logs.
hard fix·5 min read
By Vishvesh Patel·DevOps Engineer·Updated 21 September 2026
NodeCreationFailure: Instances failed to join the kubernetes cluster Nodegroup health: NodeCreationFailure kubectl get nodes No resources found failed to get token: exit status 255
Do this first3 steps
Run these in order. Each one tells you what its output means before you change anything.
• Check the node group's health message
EKS reports a specific issue code such as NodeCreationFailure, AccessDenied or Ec2SubnetInvalidConfiguration. That code narrows the cause far faster than inspecting instances by hand.
• Confirm the node role has the three required policies
AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy and AmazonEC2ContainerRegistryReadOnly are all required. A node missing the CNI policy launches and never becomes Ready, because it cannot configure networking.
• Read the bootstrap log on the instance itself
Inside, journalctl -u kubelet and /var/log/cloud-init-output.log carry the real error. This is the fastest route when the health message is generic.
The instances launch and never register. Work through these in order; the first two account for most cases.
aws eks describe-nodegroup --cluster-name prod --nodegroup-name workers --query 'nodegroup.health.issues'
1. The node IAM role
Three managed policies are required and all three matter:
The CNI one is the frequently missed member, and it produces a different symptom: the node appears and never becomes Ready.
aws iam list-attached-role-policies --role-name eks-node-role \ --query 'AttachedPolicies[].PolicyName'
2. Network path to the API server
A node must reach the cluster endpoint to register.
Private subnet with no NAT gateway:
No default route means no path out, and the bootstrap hangs until it times out.
If you are using VPC endpoints instead of NAT, nodes need several: ec2, ecr.api, ecr.dkr, the S3 gateway, sts, and logs. Missing sts is a common one, because the bootstrap calls STS to get a token and fails with failed to get token.
Private cluster endpoint:
With endpointPublicAccess: false, nodes must reach the private endpoint, which requires the cluster security group to allow them and DNS resolution inside the VPC.
aws ec2 describe-route-tables \ --filters "Name=association.subnet-id,Values=subnet-0private1" \ --query 'RouteTables[].Routes[?DestinationCidrBlock==`0.0.0.0/0`]'
3. Security groups
Nodes and the control plane must talk both ways:
The EKS-managed cluster security group covers the first two by default. A custom security group on a launch template frequently does not, and that is where this breaks.
aws eks describe-cluster --name prod \ --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId'
4. Subnet tags
The VPC CNI and load balancer controller rely on tags:
Missing the cluster tag produces Ec2SubnetInvalidConfiguration rather than a silent failure, which is at least clear.
Also check the subnet has free addresses. The VPC CNI assigns a real VPC IP to every Pod, so a /27 runs out quickly and new nodes cannot allocate ENIs:
aws ec2 describe-subnets --subnet-ids subnet-0private1 \ --query 'Subnets[].Tags'
5. The AMI and bootstrap
A custom launch template that overrides user data must still call the bootstrap script:
Replacing user data without that line gives you an instance that boots and never joins. For AL2023 nodes the mechanism is different, a NodeConfig YAML in user data rather than the shell script, and mixing the two is a common error after an AMI upgrade.
Check the AMI matches the cluster version. A node AMI more than one minor version behind the control plane is unsupported and may not register.
#!/bin/bash /etc/eks/bootstrap.sh prod
Read the instance
This is the fastest route when the health message is generic. The kubelet log usually names the exact failure: a token error, a connection timeout to the endpoint, or a missing configuration file.
SSM needs AmazonSSMManagedInstanceCore on the node role, which is worth attaching to every node group for exactly this reason.
aws ssm start-session --target i-0abc123
If aws-auth is still in use
On a CONFIG_MAP cluster the node role must be mapped, or nodes authenticate as nobody:
Managed node groups add this automatically. Self-managed nodes and Karpenter do not, and a deleted entry stops every future node joining while existing ones carry on working, which makes it look like a capacity problem.
mapRoles: |
- rolearn: arn:aws:iam::123456789012:role/eks-node-role
username: system:node:{{EC2PrivateDNSName}}
groups: [system:bootstrappers, system:nodes]A checklist
Frequently Asked Questions
Which IAM policies does an EKS node role need?
Three: AmazonEKSWorkerNodePolicy so the node can describe the cluster and register, AmazonEKS_CNI_Policy so the VPC CNI can attach ENIs and assign Pod IPs, and AmazonEC2ContainerRegistryReadOnly so it can pull images. The CNI policy is the one most often forgotten, and it produces a distinctive symptom: the node registers and appears in kubectl get nodes but stays NotReady, because networking never comes up. Attaching AmazonSSMManagedInstanceCore as well is worth it for debugging.
Why do nodes in a private subnet fail to join?
Because they have no route to the EKS API endpoint. Either add a NAT gateway with a default route, or create VPC endpoints. The endpoint list is longer than people expect: ec2, ecr.api, ecr.dkr, the S3 gateway endpoint for image layers, sts because the bootstrap exchanges an STS token, and logs for CloudWatch. A missing sts endpoint produces failed to get token, which is a useful clue when you see it.
What happens if the subnet is not tagged?
The kubernetes.io/cluster/<cluster-name> tag is used for resource discovery, and without it EKS reports Ec2SubnetInvalidConfiguration on the node group. The kubernetes.io/role/elb and kubernetes.io/role/internal-elb tags are separate and affect load balancer placement rather than node joining, so a cluster with healthy nodes can still fail to provision a LoadBalancer Service if those are missing.
Why did nodes stop joining when existing ones are fine?
On a cluster still using the aws-auth ConfigMap, the node role mapping is what lets new nodes authenticate. Deleting or corrupting that entry stops every future node joining while running nodes continue working, because they are already authenticated. The symptom looks like a capacity or scaling problem. Managed node groups maintain the entry automatically; self-managed nodes and Karpenter rely on you keeping it, which is one more reason to move to access entries.
How do I see why a specific instance failed to bootstrap?
Connect with aws ssm start-session --target <instance-id>, which needs AmazonSSMManagedInstanceCore on the node role, then read journalctl -u kubelet -n 100 and /var/log/cloud-init-output.log. The kubelet log names the actual failure rather than the generic NodeCreationFailure the node group reports: a token error points at STS or IAM, a connection timeout at the network path, and a missing config file at a broken launch template.
Reference and practice
Learn the underlying concept
Other AWS errors
Free, comprehensive DevOps tutorials and learning roadmaps. Master Docker, Kubernetes, CI/CD, and more.
Get notified about new tutorials and features.
• Roadmaps
• Interview Prep
• Certifications
• Company
© 2026 DevOpsLesson. All rights reserved.
Need to practice this scenario?
Simulate this error condition and practice recovery commands safely in our in-browser terminal.