kubectl exec, logs, and port-forward

Three commands for working with what is already running: read its output, get inside it, and reach it from your machine.

logs

kubectl logs payments-api-7d4f-x8k2
kubectl logs payments-api-7d4f-x8k2 -f              # follow
kubectl logs payments-api-7d4f-x8k2 --tail=100
kubectl logs payments-api-7d4f-x8k2 --since=15m
kubectl logs payments-api-7d4f-x8k2 --timestamps

The flag that matters most

kubectl logs payments-api-7d4f-x8k2 --previous

When a container is crash-looping, plain kubectl logs shows the instance that has just started, which during a backoff has produced nothing yet. --previous shows the run that actually failed, and that is where the error is.

If you learn one thing about debugging Pods, this is it. A CrashLoopBackOff with apparently empty logs is nearly always someone forgetting --previous.

Multiple containers

kubectl logs <pod> -c api                  # one container
kubectl logs <pod> --all-containers        # all of them

With more than one container in a Pod, kubectl logs refuses to guess and errors until you pick one.

Across Pods

kubectl logs -l app=payments-api --tail=50 --prefix
kubectl logs -l app=payments-api -f --max-log-requests=10

--prefix labels each line with its Pod name, which is essential when reading three replicas at once. Following many Pods is capped by --max-log-requests (default 5).

For serious multi-Pod log reading, stern remains better than anything built in.

What logs are and are not

kubectl logs reads files on the node written by the container runtime from the container's stdout and stderr. Two consequences:

• An application logging to a file inside the container produces nothing. Log to stdout.
• Logs are deleted with the Pod. --previous reaches back exactly one container generation, not further. Anything you need to keep requires a log shipper.

exec

kubectl exec -it payments-api-7d4f-x8k2 -- sh
kubectl exec -it payments-api-7d4f-x8k2 -c api -- bash
kubectl exec payments-api-7d4f-x8k2 -- env        # single command, no TTY
kubectl exec payments-api-7d4f-x8k2 -- ls /config

The -- separator matters. Everything after it goes to the container, so without it flags like -l are interpreted by kubectl instead.

Use sh rather than bash by default. Most slim images have no bash.

Things worth checking from inside a container:

kubectl exec <pod> -- env | sort                       # what config did it actually get?
kubectl exec <pod> -- cat /config/app.yaml             # is the ConfigMap mounted?
kubectl exec <pod> -- wget -qO- localhost:8080/healthz # does the probe endpoint work?
kubectl exec <pod> -- nslookup payments-db             # does DNS resolve?

That first one settles a great many arguments. Checking what the container actually received, rather than what the manifest says, is usually faster than reasoning about the manifest.

Distroless images have no shell

kubectl exec needs a shell inside the container. Distroless and scratch images deliberately have none:

The answer is an ephemeral debug container. A temporary container added to the running Pod, sharing its network namespace:

kubectl debug -it payments-api-7d4f-x8k2 \
  --image=nicolaka/netshoot \
  --target=api

--target=api shares the target container's process namespace so you can see its processes. nicolaka/netshoot carries curl, dig, tcpdump, netstat and most of what you would want.

The Pod is unchanged when you exit. Nothing is rebuilt, nothing restarts.

To debug a Pod that will not start at all, copy it with a different command:

kubectl debug payments-api-7d4f-x8k2 -it --copy-to=debug-pod --container=api -- sh

And for a node:

kubectl debug node/node-1 -it --image=nicolaka/netshoot

port-forward

kubectl port-forward pod/payments-api-7d4f-x8k2 8080:8080
kubectl port-forward service/payments-api 8080:80
kubectl port-forward deployment/payments-api 8080:8080

Tunnels a local port through the API server to something in the cluster. This is the right way to reach an internal service from a laptop, no Ingress, no NodePort, no exposing anything.

# Reach a database locally
kubectl port-forward -n payments service/postgres 5432:5432
psql -h localhost -U payments

Forwarding to a Service picks one Pod and stays with it; it is not load balanced. Forwarding to a Deployment picks one of its Pods.

Notes worth knowing: 0.0.0.0:8080:8080 as the address binds all interfaces rather than just localhost, the connection drops when the Pod is replaced, and it only handles TCP.

Copying files

kubectl cp payments-api-7d4f-x8k2:/var/log/app.log ./app.log
kubectl cp ./fixture.json payments-api-7d4f-x8k2:/tmp/fixture.json
kubectl cp <pod>:/path ./local -c api          # pick a container

kubectl cp requires tar inside the container, so it fails on distroless images just as exec does.

A debugging order that works

1. kubectl get pods. What state is it in?
2. kubectl describe pod <name>. Do the events explain it?
3. kubectl logs <name> --previous. What did the failing run say?
4. kubectl exec <name> -- env. Did it get the configuration you expected?
5. kubectl debug, if there is no shell, or you need network tools.

Most problems are answered by steps 2 and 3.

Quick reference

Key takeaways

• kubectl logs --previous is the crash-debugging command. Plain logs shows the container that just started.
• Only stdout and stderr are captured. An application logging to a file produces nothing.
• Logs die with the Pod, and --previous reaches back exactly one generation.
• -- separates kubectl's flags from the container's command.
• kubectl exec <pod> -- env answers most "why is it misconfigured" questions immediately.
• Distroless images have no shell: use kubectl debug with an ephemeral container.
• port-forward is how you reach an internal Service without exposing it.

Check your understanding

<MCQExercise exercise={{ id: 'k8s-execlogs-previous', title: 'Empty logs on a crash loop', description: 'A Pod is in CrashLoopBackOff. kubectl logs returns nothing at all. What is the explanation?', options: [ { id: 'a', text: 'Plain kubectl logs reads the container starting now, which has produced no output yet, --previous reads the run that crashed' }, { id: 'b', text: 'Logs are unavailable for Pods in CrashLoopBackOff' }, { id: 'c', text: 'The container writes to a log file, so nothing reaches stdout' }, { id: 'd', text: 'The log files were rotated away by the container runtime' }, ], correctId: 'a', explanation: 'During a backoff Kubernetes waits before the next attempt, so the current container instance is either not started or only moments old and has written nothing. The output you want belongs to the terminated instance, which --previous retrieves. This is the single most common stumbling block when debugging crash loops.', wrongExplanations: { b: 'Logs remain fully available in that state. It is which container instance you are reading that changes.', c: 'This is a real and separate problem worth ruling out, but it would give empty output consistently rather than only during a crash loop, and --previous would be empty too.', d: 'Rotation applies to long-running containers producing large volumes. A container that dies seconds after starting has nothing to rotate.', }, }} />

<MCQExercise exercise={{ id: 'k8s-execlogs-distroless', title: 'No shell in the container', description: 'kubectl exec into a distroless production container fails with "exec: sh: executable file not found in $PATH". What is the right approach?', options: [ { id: 'a', text: 'Use kubectl debug to attach an ephemeral container with a shell and tooling, sharing the Pod's namespaces' }, { id: 'b', text: 'Rebuild the image on a base that includes a shell and redeploy it' }, { id: 'c', text: 'Use kubectl cp to copy a shell binary into the container first' }, { id: 'd', text: 'Use kubectl logs, since exec is impossible for distroless images' }, ], correctId: 'a', explanation: 'kubectl debug adds a temporary container to the running Pod with an image of your choosing, sharing its network namespace and, with --target, its process namespace. You get curl, dig and tcpdump against the live workload without changing the image, restarting anything, or weakening the security properties distroless was chosen for.', wrongExplanations: { b: 'This works but discards the running state you were trying to inspect, requires a deploy, and permanently enlarges the attack surface of a production image.', c: 'kubectl cp itself needs tar inside the container, which distroless images also lack, so the copy fails before you get anywhere.', d: 'Logs are useful but limited to what the application chose to print. Ephemeral containers make exec-style debugging entirely possible.', }, }} />