Debugging & Troubleshooting

In this exercise, you'll inspect three Deployments and repair two that are failing. Use Pod status, Events, and logs to identify what went wrong.

Type commands in the browser terminal (on a phone, use the Terminal tab). Use the Pod names from your output wherever an example has a placeholder. Reset restarts this exercise, and help lists supported commands.

This is a simplified simulation. The replacement images below demonstrate repairing a Pod template; swapping a real API for nginx would not repair that API's functionality. At work, use your application's known-good image and verify actual requests.

Survey the cluster

Start with a high-level view:

kubectl get deployments

nginx-deployment is ready. Both api-server and worker-service show 0 ready replicas. Inspect their Pods:

kubectl get pods

Notice the STATUS column. Instead of Running, you'll see:

• ImagePullBackOff: Kubernetes can't pull the container image.
• CrashLoopBackOff: the container starts but keeps crashing.

Diagnosing ImagePullBackOff

The api-server pods are stuck in ImagePullBackOff. This usually means the image name is wrong, the registry doesn't exist, or authentication is missing.

Use describe to see what happened:

kubectl describe pod <api-server-pod-name>

Look at the Events section at the bottom. You'll see Failed and BackOff events with messages about the image pull failure. The image nonexistent-registry.io/api-server:2.0 doesn't exist.

Fix it by updating the image to something that works:

kubectl set image deployment/api-server api-server=nginx:1.26.2

Watch the pods recover:

kubectl get pods

The old pods will terminate and new ones will start with the corrected image.

Diagnosing CrashLoopBackOff

The worker-service pod is in CrashLoopBackOff. This means the container starts but the process inside it keeps failing.

First, check the pod details:

kubectl describe pod <worker-service-pod-name>

The Events section shows BackOff warnings. But to understand why it's crashing, we need the logs:

kubectl logs <worker-service-pod-name>

The logs show that Node.js can't find the application's entry point. The image node:20-alpine-broken is faulty.

Fix it with a working image:

kubectl set image deployment/worker-service worker-service=node:20-alpine

Using kubectl exec

Now that you've fixed the api-server deployment, its pods should be running. For running containers, kubectl exec lets you inspect the container from inside. Try it on one of the api-server pods you just fixed:

kubectl exec <api-server-pod-name> -- hostname
kubectl exec <api-server-pod-name> -- cat /etc/hostname

Use exec to inspect configuration files, environment variables, and the filesystem inside a running container. During CrashLoopBackOff, the container may exit before you can connect. Check its logs and Events first.

Understanding Events

Kubernetes records events for significant occurrences. View all cluster events:

kubectl get events

Events include scheduling decisions, image pulls, container starts, failures, and more. They're timestamped and categorized as Normal or Warning.

When debugging:

1. Events tell you what happened
2. Logs tell you why it happened
3. Describe gives you the full picture

Debugging checklist

When something isn't working:

1. kubectl get pods: check Pod status and ready counts.
2. kubectl describe pod <name>: look at Events and container State.
3. kubectl logs <name>: read application output.
4. kubectl get events: see activity in the current namespace.
5. kubectl exec <name> -- <command>: inspect a running container.