context deadline exceeded

KubernetesKubernetes

Fix Kubernetes "context deadline exceeded": find which component timed out, then check admission webhooks, API server and etcd, CSI calls, probes and kubectl.

hard fix·6 min read

By Vishvesh Patel·DevOps Engineer·Updated 21 September 2026

Run these in order. Each one tells you what its output means before you change anything.

• Identify which component produced the message

The prefix is everything. "failed calling webhook" is admission, "Liveness probe failed" is the kubelet, "rpc error" is a CSI or device plugin gRPC call, and a bare kubectl timeout is the API server or your own connection.

• If it mentions a webhook, find out whether its backend is alive

A webhook whose Service has no ready endpoints blocks every matching API request until it times out. This is the single most common cause of a cluster that suddenly cannot create anything.

• Measure API server latency before blaming the network

Slow etcd shows up here first and explains cluster-wide timeouts. If these look healthy, the problem is specific to one component rather than the control plane.

context deadline exceeded is Go's standard timeout error. Kubernetes is written in Go, so almost every component can emit it, and on its own it tells you only that something waited and gave up.

The prefix is the diagnosis. Read it before anything else.

Every create and update of a matching resource goes through the webhook. If its backend is down, the API server waits timeoutSeconds and then applies failurePolicy. With failurePolicy: Fail, the request is rejected, and nothing matching that webhook can be created cluster-wide.

This is the classic way a cluster becomes unable to schedule anything: the webhook's own Pods get evicted, and now they cannot be recreated because the webhook that must admit them is down.

Empty ENDPOINTS confirms it.

To break the deadlock, delete the webhook configuration, fix the backend, then reinstall it:

Save it first. Deleting without a copy loses the CA bundle, which is not always easy to regenerate.

Two things worth designing for:

Scope the webhook with namespaceSelector so it never intercepts kube-system or its own namespace:

Use failurePolicy: Ignore for anything non-critical. A policy webhook that annotates Pods should not be able to stop the cluster. Reserve Fail for genuine security controls where allowing an unchecked object through is worse than an outage.

A bare timeout from kubectl on many different commands points at the control plane:

etcd is the usual constraint, and it is disk-latency sensitive far more than CPU. fsync durations above about 25ms produce visible cluster-wide slowness. Common causes: etcd on network-attached storage rather than local NVMe, a large number of Events, or oversized objects.

Tens of thousands of Events is a real load. They expire after an hour by default, so a persistently large count means something is generating them fast.

Also check for very large objects. A ConfigMap near the 1MiB limit being read frequently by a controller is a surprisingly effective way to slow etcd down.

This is a gRPC call to a driver that did not answer. Look at the driver, not Kubernetes:

Cloud API rate limiting is a frequent cause on large clusters. Many volumes being attached at once exhausts the provider's request quota and calls start timing out, which looks like a Kubernetes fault and is not.

Covered in depth on the readiness probe page, but the short version: timeoutSeconds defaults to 1 second, which a health endpoint that touches a database will exceed under load. It then fails for every replica at once.

If only your machine is affected:

A private API endpoint reached over a VPN, or a long path to another region, produces timeouts that look like cluster problems and are not. -v=6 prints the request duration, which settles it quickly.

For genuinely large collections, paginate rather than raising the timeout:

What does "context deadline exceeded" actually mean?

It is Go's generic timeout error, produced when an operation carrying a deadline runs out of time. Kubernetes components are written in Go, so the API server, kubelet, controllers and CSI drivers can all emit it, and the phrase itself carries no information about what was slow. The text before it does: failed calling webhook is admission, Liveness probe failed is the kubelet, and rpc error: code = DeadlineExceeded is a gRPC call to a plugin. Identify the component before investigating anything.

Why did a broken admission webhook stop me creating any Pods?

Because every matching create and update goes through the webhook, and with failurePolicy: Fail the API server rejects the request when the webhook does not answer. If the webhook's own Pods are down, you get a deadlock: the Pods cannot be recreated because the webhook that must admit them is unavailable. The way out is to save the webhook configuration to a file, delete it, recover the backend and reapply it. Scoping webhooks with a namespaceSelector that excludes kube-system and their own namespace prevents the deadlock forming.

How do I tell whether the API server or my network is slow?

Run kubectl get nodes -v=6, which prints the round-trip duration for each request. If that is fast while the cluster is otherwise unhappy, your connection is fine. For the control plane, kubectl get --raw /readyz?verbose reports each component's health, and the etcd_request_duration_seconds histogram in /metrics shows whether etcd is the constraint. etcd is unusually sensitive to disk latency, so fsync times above roughly 25 milliseconds produce cluster-wide slowness even when CPU looks fine.

Can too many Events slow down my cluster?

Yes. Events are stored in etcd like any other object, and a cluster generating them rapidly, typically from a Pod crash-looping at scale or a controller reconciling in a tight loop, puts real write pressure on it. They expire after an hour by default, so a persistently high count from kubectl get events -A --no-headers | wc -l means something is producing them continuously. Fixing the source matters more than tuning retention, since the write load is the problem rather than the storage.

Should I set failurePolicy to Ignore on all my webhooks?

No, but you should default to it and make Fail a deliberate choice. Fail is correct for a security control where admitting an unchecked object is worse than refusing the request, such as a policy that blocks privileged containers. For anything that annotates, defaults or labels, Ignore means a webhook outage degrades a convenience rather than stopping the cluster. Whichever you choose, keep timeoutSeconds low, since the API server waits the full timeout on every matching request before applying the policy.

Free, comprehensive DevOps tutorials and learning roadmaps. Master Docker, Kubernetes, CI/CD, and more.

Get notified about new tutorials and features.

• Roadmaps

• Interview Prep

• Certifications

• Company

© 2026 DevOpsLesson. All rights reserved.