pal-e-docs

TODO: Plan-and-Apply-Before-Merge Convention

todo-plan-apply-before-merge Todo

todo done
Project

TODO: Plan-and-Apply-Before-Merge Convention

Problem

Terraform PRs get merged before the infrastructure is proven live. When tofu apply fails post-merge, each fix requires a separate PR. This wastes context, pollutes git history, and risks deploying broken infra.

Observed (2026-03-14, Phase 5a Keycloak deploy — 5 fixes on 1 branch)

Lucas caught this: "it would be better to plan and apply before merging so if we run into errors we can clean PR." Five issues were found and fixed in-place on the PR branch:
  • Bitnami Docker Hub images gone — entire Bitnami container registry removed. Switched from Helm chart to raw k8s manifests with official quay.io/keycloak/keycloak:26.0.7.
  • PVC WaitForFirstConsumer timeout — local-path provisioner doesn't bind PVCs until a pod mounts them. Added wait_until_bound = false.
  • OOMKilled at 1Gi — Keycloak JVM needs ~1.5Gi during startup (schema init + class loading). Bumped to 2Gi limit.
  • Health probe 404 — Keycloak requires KC_HEALTH_ENABLED=true to expose health endpoints. Not enabled by default.
  • Probes on wrong port + liveness killing during startup — Keycloak 26.x serves health on management port 9000, not application port 8080. Liveness probe was killing the pod before JVM finished starting. Fixed with port 9000 + startup probe (30s delay, 30 retries at 10s = 5 min window).
Without plan-and-apply-before-merge, this would have been 5 separate PRs instead of 5 commits on one branch. Final merge is a proven, battle-tested deployment.

Proposed Convention

For any PR that changes Terraform/infrastructure resources:
  • Dev agent creates PR — code review + QA as normal
  • Betty Sue runs tofu plan from the PR branch (worktree) — verify plan output is clean
  • Betty Sue runs tofu apply -target from the PR branch — deploy the specific resources
  • Verify live — kubectl get pods, hit the URL, check health endpoints
  • Fix issues on the PR branch — push additional commits, re-apply
  • Merge only after infra is proven live

Scope for Phase

  • Write a convention-plan-apply-before-merge note with the full workflow
  • Update pr-lifecycle SOP to include a "Terraform gate" step between QA approval and merge
  • Update template-issue to include a ### Terraform Changes section when applicable
  • Consider a PreToolUse hook on mcp__forgejo__merge_approved_pr that warns if the PR touches *.tf files and no tofu apply has been run
  • Document the -target pattern for scoped applies (avoid touching unrelated resources)
  • Document the force-unlock recovery pattern for stale state locks
  • Document the tofu state rm + tofu import recovery for state/reality drift
  • plan-pal-e-agency — future phase
  • pr-lifecycle — needs update
  • plan-pal-e-platform — Keycloak deploy was the triggering incident
  • PR #34 on pal-e-platform — the 5-commit proof