pal-e-docs

Phase: CI — State Backup CronJob

phase-pal-e-platform-ci-6-1-state-backup Phase

Project
Goal: Both Terraform state secrets backed up daily to MinIO. Documented restore procedure. Safety net before CI pipeline goes live.
Owner: Dev agent
Repo: forgejo_admin/pal-e-platform
Depends on: None — independently executable

Why

DORA impact: Directly improves MTTR. If state is corrupted or lost during a bad apply, recovery without backups means manual state reconstruction — hours of work. With daily backups to MinIO, recovery is a restore command. This is the safety net that makes Phase 6.4 (apply-on-merge) safe to deploy.

Scope

1. MinIO bucket + IAM. Create tf-state-backups bucket via Terraform. Create MinIO IAM user tf-backup with policy scoped to the bucket. Deploy credentials as a k8s Secret in tofu-state namespace.
2. CronJob. Kubernetes CronJob in tofu-state namespace that:
  • Runs daily at 02:00 UTC
  • Reads tfstate-default-pal-e-platform and tfstate-default-pal-e-services secrets
  • Base64-decodes the tfstate key from each
  • Uploads to s3://tf-state-backups/{repo}-{date}.json via MinIO (using mc or curl)
  • Retains last 30 days (delete older backups in the same script)
3. Deploy via Terraform. All resources (bucket, IAM, Secret, CronJob) defined in terraform/main.tf.

Deliverables

  • PR #39 merged (2026-03-14) — Closes #36
  • 9 TF resources: MinIO bucket tf-state-backups, IAM user tf-backup + policy + attachment, k8s Secret, ServiceAccount, Role + RoleBinding (scoped to 2 state secrets), CronJob
  • CronJob: daily 02:00 UTC, reads state secrets, base64-decodes, uploads to MinIO, prunes >30 days
  • RBAC: resource_names constraint limits access to only tfstate-default-pal-e-platform and tfstate-default-pal-e-services
  • QA nits: runtime mc download (external dep), bitnami/kubectl:1.31 pin needs tracking, no force_destroy on bucket (consistent with existing patterns)
  • phase-pal-e-platform-ci-hardening — parent phase (CI Pipeline & Team Hardening)
  • plan-pal-e-platform — grandparent plan (Platform Hardening)