pal-e-docs

SOP: Platform Terraform Changes

sop-platform-tf-changes Sop

sop active
Project

SOP: Platform Terraform Changes

Purpose

Document the standard workflow for making infrastructure changes via Terraform in the pal-e-platform and pal-e-services repos. All changes go through CI — no manual tofu apply allowed (except break-glass).

Prerequisites

  • Access to Forgejo (forgejo_admin/pal-e-platform, ldraney/pal-e-services)
  • Woodpecker CI activated on both repos
  • Branch protection enabled on main — direct push blocked

Standard Workflow

pal-e-platform (CI-driven apply, individual TF_VAR_* secrets)

  • Create Forgejo issue from plan phase or bug note
  • Create branch from issue (naming: {issue-num}-{slug})
  • Make changes in worktree or local branch
  • Local validation: cd terraform && tofu init && tofu fmt -check && tofu validate
  • Local plan (optional): tofu plan -lock=false. The -lock=false flag is REQUIRED when running from a worktree or any non-CI context.
  • Push and create PR — body MUST include Closes #N
  • CI runs automatically: validate + plan on PR, plan output posted as PR comment
  • Review plan comment on the PR — verify expected changes
  • Merge PR — triggers apply step on main
  • Verify — check Woodpecker pipeline success, confirm resources in cluster

pal-e-services (CI-driven apply, bundled tfvars_content secret)

k3s.tfvars is gitignored (contains sensitive config). Instead of individual TF_VAR_* secrets, all tfvars are bundled into a single base64-encoded tfvars_content Woodpecker secret. CI decodes it at runtime.
  • Create Forgejo issue from plan phase or bug note
  • Create branch from issue (naming: fix/{issue-num}-{slug})
  • Make terraform changes in worktree (services.tf, variables.tf, etc.)
  • Edit ~/secrets/pal-e-services/k3s.tfvars with new variable values
  • Update tfvars_content secret: 
    base64 -w0 ~/secrets/pal-e-services/k3s.tfvars
mcp__woodpecker__update_repo_secret(repo_full_name="ldraney/pal-e-services", name="tfvars_content", value=<base64 output>)
  • Local plan (optional): Symlink k3s.tfvars into clone, run tofu plan -lock=false -var-file=k3s.tfvars
  • Push and create PR — body MUST include Closes #N
  • CI runs automatically: validate + plan on PR, plan output posted as PR comment
  • Review plan comment on the PR — verify expected changes
  • Merge PR — triggers apply step on main
  • Verify — check resources in cluster (kubectl get secret, kubectl get ns, ArgoCD UI)
Critical: Step 5 (updating tfvars_content) must happen before pushing the branch, otherwise CI plan will fail because it won't have the new variables. See sop-secrets-management for the full procedure.

pal-e-deployments (CI validation on PR)

Kustomize overlays are validated in CI before merge:
  • Push PR — Woodpecker CI triggers automatically
  • CI validates: kubectl kustomize on all changed overlays + kubectl apply --dry-run=server for server-side schema validation
  • QA reviews — code review + CI green
  • Merge — ArgoCD picks up the overlay on next sync

What NOT to Do

  • No manual tofu apply — all applies go through CI on merge to main (except break-glass)
  • No direct push to main — branch protection is enforced
  • No --force push — creates state drift risk
  • No skipping plan review — always verify the plan comment before merging
  • No tofu plan without -lock=false from worktrees — a locked plan from a worktree blocks CI apply on ALL branches
  • No forgetting to update tfvars_content when editing k3s.tfvars for pal-e-services — CI will use stale values

Break-Glass Procedure

If CI is broken and an emergency change is needed:
  • SSH to node
  • cd ~/pal-e-platform/terraform (or ~/pal-e-services/terraform)
  • For pal-e-platform: make tofu-secrets (renders Salt pillar secrets to secrets.auto.tfvars)
  • For pal-e-services: symlink ~/secrets/pal-e-services/k3s.tfvars into terraform dir
  • tofu init && tofu plan (review carefully)
  • tofu apply (only after plan review)
  • File an issue immediately to fix CI
  • Push the change through a PR retroactively

CI Pipeline Details

pal-e-platform

  • Pipeline file: .woodpecker.yaml
  • Secrets: 17 Woodpecker repo secrets (TF_VAR_* pattern)
  • Image: ghcr.io/opentofu/opentofu:1.9 (all steps)
  • Events: validate + plan on pull_request, apply on push to main

pal-e-services

  • Pipeline file: .woodpecker/terraform.yaml
  • Secrets: 3 Woodpecker repo secrets (kubeconfig_content, forgejo_token, tfvars_content)
  • Image: ghcr.io/opentofu/opentofu:1.9 (all steps)
  • Events: validate + plan on pull_request, apply on push to main
  • Secret delivery: tfvars_content is base64-decoded to /tmp/k3s.tfvars at runtime, then passed via -var-file

DORA Baseline (2026-03-14)

  • Deployment Frequency: ~1 deploy/day (repo just activated, single active contributor)
  • Lead Time for Changes: Not yet measurable (Woodpecker API doesn't expose timestamps)
  • Change Failure Rate: 0% on main applies (1/1 successful), 37.5% on PR pipelines (expected — initial CI setup iteration)
  • MTTR: No production failures yet to measure
  • Note: These are bootstrap numbers from day 1. Re-measure after 2 weeks of steady-state operation.
  • plan-pal-e-platform — Platform Hardening plan
  • .woodpecker.yaml in pal-e-platform — CI pipeline definition
  • .woodpecker/terraform.yaml in pal-e-services — CI pipeline definition
  • sop-secrets-management — how secrets reach CI (including tfvars_content update procedure)
  • deployment-lessons — operational lessons learned