pal-e-docs

TF: Rollback Strategy + Disaster Recovery

tf-rollback-strategy Doc

sre terraform roadmap
Project

Terraform Rollback Strategy + Disaster Recovery

The Problem

In the last 48 hours, both Grafana and pal-e-docs crashed with no automated rollback. Recovery required Lucas to be at his laptop, diagnose the issue, and manually run tofu apply or fix Helm values. This is the motivation for moving from Litestream/SQLite to PostgreSQL — but the infrastructure itself also needs rollback capability.

Rollback Layers

There are three distinct rollback surfaces in our system:
Layer What Breaks Current Recovery Target Recovery
<strong>Application code</strong> Bad deploy of pal-e-docs, basketball-api ArgoCD self-heal reverts to last good k8s manifest. Image Updater can be paused. Already good — ArgoCD handles this. Add <code>argocd app rollback</code> runbook.
<strong>Helm values</strong> Bad Terraform change to a Helm release (e.g., Grafana password change, resource limits) Manual: revert git commit, re-run <code>tofu apply</code> CI pipeline: revert PR, auto-apply on merge
<strong>Infrastructure</strong> Namespace deleted, PVC destroyed, state corrupted Manual: import resources, recreate from scratch State backups + documented recovery runbooks

Specific Rollback Mechanisms

1. Helm Release Rollback (most common failure)

Helm tracks release history. Even when managed by Terraform, you can:
Warning: Helm rollback + Terraform = drift. Always reconcile with tofu apply after a manual Helm rollback. This is a stop-gap, not a process.

2. Git Revert + Re-Apply (target process)

With a CI pipeline, rollback is:
  • Create a revert PR: git revert <bad-commit>
  • PR gets auto-planned, reviewed, merged
  • Pipeline runs tofu apply with reverted state
This is the correct process. It requires the CI pipeline (see tf-pipeline-design).

3. State Backup

Kubernetes backend stores state in secrets in tofu-state namespace. These should be backed up:
Should be a CronJob in-cluster.

4. PVC Disaster Recovery

If a PVC is destroyed (Forgejo data, Prometheus metrics, Grafana dashboards):
  • Grafana: Dashboards are ConfigMaps (recoverable via TF). Only loss is non-TF dashboards created in UI.
  • Prometheus: Metrics history lost, but scraping resumes immediately. Acceptable for 15d retention window.
  • Forgejo: Critical — all git repos live here. Needs backup strategy (Forgejo dump CronJob or MinIO backup).
  • PostgreSQL: Moving pal-e-docs here specifically for better backup/restore (pg_dump vs Litestream).

Incident Response Runbook (Today)

  • Check pod status: kubectl get pods -A | grep -v Running
  • Check events: kubectl get events -A --sort-by=.lastTimestamp | tail -20
  • Describe failing pod: kubectl describe pod <name> -n <ns>
  • Check logs: kubectl logs <pod> -n <ns> --previous (for crashed pods)
  • If Helm values issue: fix in TF, run tofu apply -var-file=k3s.tfvars
  • If urgent: helm rollback <release> -n <ns> then reconcile TF later

What We Need

  • State backup CronJob (daily to MinIO)
  • Forgejo backup CronJob (daily dump to MinIO)
  • CI pipeline for fast git-revert-based rollback
  • Alerting (see observability plan) so we KNOW something crashed before a user reports it