pal-e-docs

Bug: Grafana CrashLoopBackOff — Duplicate Default Datasource

bug-grafana-crashloop Todo

bug resolved
Project

Problem

Grafana pod in monitoring namespace in CrashLoopBackOff with 938+ restarts.

Root Cause

The kube-prometheus-stack Helm chart creates a default Prometheus datasource. The Terraform-managed grafana-loki-datasource ConfigMap also creates a Loki datasource. Grafana defaults isDefault to true when omitted, creating two defaults, which Grafana rejects at startup.

Fix

Set isDefault = false on the Loki datasource in pal-e-platform/terraform/main.tf. PR #28 merged.

Debugging Story

  • kubectl logs --tail=30 — found the exact error message in one command
  • Traced to Terraform — two resources both creating datasources: helm_release.kube_prometheus_stack (Prometheus, isDefault=true) and kubernetes_config_map_v1.grafana_loki_datasource (Loki, isDefault omitted → defaults to true)
  • Fixed in Terraform — added isDefault = false to Loki datasource
  • Chicken-and-egg during apply — tofu apply tried to update the ConfigMap AND reconcile the Helm release, but the Helm release was stuck waiting for Grafana to be healthy, which couldn't happen until the ConfigMap was fixed
  • Unblocked with kubectl — patched the ConfigMap directly via kubectl apply, deleted the crashing pod, Grafana started cleanly, then tofu apply completed
  • Verified — kubectl exec into Grafana pod, confirmed both datasource provisioning files present with correct isDefault values

Lessons

  • Always check kubectl logs --previous for CrashLoopBackOff — the crash reason is usually in the last log lines
  • Helm chart datasource provisioning and manually-managed ConfigMap datasources can conflict on isDefault
  • When Terraform is blocked by a resource it's trying to fix, sometimes you need to break the cycle with a direct kubectl patch, then let Terraform converge

Impact

No Grafana UI for ~5 days. Prometheus and Loki were collecting data the entire time — no data loss.

Plan

No plan required — quick config fix.

Acceptance Criteria

  • [x] Grafana pod is Running (not CrashLoopBackOff)
  • [x] Both Prometheus and Loki datasources available in Grafana
  • [x] Grafana dashboards load successfully