pal-e-docs

Phase 17b: Terraform State Governance

phase-platform-17b-tf-state-governance Phase

phase
Project
Goal: Prevent stale state locks from blocking CI apply-on-merge pipelines, and establish governance for multi-project terraform state as the platform scales.
Owner: Dev agent
Repo: forgejo_admin/pal-e-platform
Depends on: phase-pal-e-platform-ci-6-4-apply-on-merge (COMPLETED), phase-platform-17a-woodpecker-secrets (COMPLETED)

Why

Pipeline #80 failed on main because a previous tofu apply (pipeline #76) crashed mid-execution and left an orphaned state lock in the Kubernetes backend. Every subsequent merge-triggered apply failed with "state is already locked" until manual tofu force-unlock. The Kubernetes backend supports locking (via lease objects) but has no TTL — crashed applies hold the lock forever. With two TF projects today and more coming, this is a scaling hazard.

Scope

17b.1: CI Lock Recovery (immediate)
  • Add lock-aware retry to the apply step in .woodpecker.yaml: if tofu apply fails with "state is already locked", extract the lock ID, run tofu force-unlock -force, and retry once
  • Add a hard timeout to the apply step so crashed applies don't hold locks indefinitely
  • Apply the same pattern to any future TF project CI pipelines
17b.2: State Hygiene SOP
  • Document manual unlock procedure: how to identify stale locks, when it's safe to force-unlock, when it's NOT safe (concurrent legitimate applies)
  • Add to sop-ci-pipeline-recovery as a subsection
  • Add pipeline #80 incident to deployment-lessons
17b.3: Multi-Project TF Governance (future, when project count > 3)
  • Evaluate remote backend with native lock TTL (Consul, S3+DynamoDB, or PostgreSQL) as alternative to Kubernetes backend
  • Document state isolation conventions: one secret_suffix per project, all in tofu-state namespace
  • Consider shared TF module patterns if services terraform grows
Current state: Two TF projects, both using backend "kubernetes" in tofu-state namespace. pal-e-platform has CI apply-on-merge (high lock risk). pal-e-services is manual-only (low risk).

Deliverables

  • 17b.1 CI Lock Recovery (COMPLETED) — PR #100 merged. Lock-aware retry in .woodpecker.yaml apply step. Auto-detects stale state locks, extracts lock ID, force-unlocks, retries once. POSIX sh compatible.
  • 17b.2 State Hygiene SOP (COMPLETED) — Added State Lock Recovery section to sop-ci-pipeline-recovery. Added pipeline #80 incident to deployment-lessons.
  • 17b.3 Multi-Project TF Governance (FUTURE) — Evaluate remote backend with native lock TTL when project count exceeds 3.
  • plan-pal-e-platform — parent plan
  • phase-pal-e-platform-ci-6-4-apply-on-merge — the pipeline this hardens
  • deployment-lessons — state lock incident to be documented here
  • sop-ci-pipeline-recovery — SOP to extend with unlock procedure
  • todo-service-onboarding-validation — related onboarding hardening