pal-e-docs

SRE Debugging with kubectl

sre-kubectl-debugging Doc

active sre guide
Project

SRE Debugging with kubectl

Standard workflow for diagnosing pod failures on the pal-e k3s cluster. Each step answers a specific question in the debugging chain.

Step 1: What state is the pod in?

What this tells you: Pod phase (Running, CrashLoopBackOff, ImagePullBackOff, Pending), restart count, which node it's on, and the Events section shows WHY it's in that state (failed health check, OOM killed, image not found, etc.).
Key fields in describe:
  • State / Last State — current and previous container state, including exit codes
  • Restart Count — how many times the container restarted
  • Events — chronological log of scheduler decisions, image pulls, probe failures
  • Reason: OOMKilled — container exceeded its memory limit
  • Reason: CrashLoopBackOff — container exits repeatedly, kubelet backs off restart interval

Step 2: What is the pod saying?

What this tells you: Application-level errors. The --previous flag is essential for crash loops because the current container may not have logged anything useful yet before crashing again.
Real example (Grafana bug): kubectl logs ... --tail=30 showed "Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default" — the exact root cause in one command.

Step 3: What config does the container see?

What this tells you: Whether ConfigMaps, Secrets, and environment variables are actually mounted and visible inside the running container. Terraform may update a ConfigMap object, but the pod won't see the change until the volume is refreshed or the pod restarts. This verifies the full chain: Terraform → k8s object → pod filesystem.

Step 4: Is the service responding internally?

What this tells you: Whether the application is actually serving requests, independent of any networking (Service, Ingress, Tailscale funnel). If this fails, the problem is the application. If this works but external access fails, the problem is networking/routing.

Step 5: What does the k8s object say?

What this tells you: The actual state of k8s objects as the API server sees them. Useful when Terraform says it applied something but the pod still has old config — the ConfigMap object may not match what's mounted.

Step 6: Force a restart

When to use: After updating a ConfigMap or Secret that a pod has already cached. ConfigMaps mounted as volumes eventually refresh (~60s), but environment variables from Secrets require a pod restart.

Debugging Chain Summary

Platform-Specific Notes

  • Grafana pod has 3 containers: grafana, grafana-sc-dashboard (sidecar), grafana-sc-datasources (sidecar). Use -c grafana to target the main container.
  • Tailscale funnels run as separate proxy pods in the tailscale namespace (e.g., ts-grafana-funnel-xxx). If a funnel URL doesn't resolve, check these pods.
  • Secrets are base64-encoded in k8s. Always pipe through base64 -d when reading via jsonpath.
  • ArgoCD Image Updater caches Harbor credentials at startup. If a robot account is recreated, the Image Updater pod must be restarted to pick up new creds.