pal-e-docs

TF: CI/CD Pipeline Design + DORA Metrics

tf-pipeline-design Doc

ci-cd terraform roadmap dora
Project

Terraform CI/CD Pipeline Design

Current State: Laptop-Only

Today, all tofu plan and tofu apply runs happen from Lucas's laptop. This means:
  • No audit trail beyond git log
  • No plan review before apply
  • No automated validation (fmt, validate)
  • No way for a second developer to safely run apply
  • No rollback mechanism besides manually reverting and re-applying
  • MTTR depends on laptop availability

Target Pipeline (Woodpecker CI)

We already have Woodpecker CI. The Terraform pipeline should run there, not in GitHub Actions.

On PR (plan only)

On Merge to Main (apply)

Challenges Specific to Our Setup

Challenge Why It's Hard Approach
Kubeconfig access Woodpecker agents need cluster access to run tofu Mount kubeconfig as Woodpecker secret, or use service account token
Secrets in tfvars k3s.tfvars has plaintext passwords Woodpecker secrets → env vars → <code>TF_VAR_*</code> pattern
Provider connectivity Tailscale, MinIO, Harbor, ArgoCD providers need network access to their APIs Woodpecker agent runs in-cluster, has ClusterIP access. Tailscale provider needs OAuth (Woodpecker secret).
State locking Kubernetes backend locks via ConfigMap lease Already supported — just need to ensure pipeline doesn't run concurrent applies
Two-repo ordering pal-e-services depends on pal-e-platform resources Separate pipelines. Platform applies first. Cross-repo trigger or manual gate.
Plan output on PR Need Woodpecker to post plan as PR comment on Forgejo Use Forgejo API to post comment, or Woodpecker plugin

DORA Metrics

Once the pipeline exists, we can measure:
Metric What It Measures How to Capture
<strong>Deployment Frequency</strong> How often we deploy to production Count of successful <code>tofu apply</code> runs per week
<strong>Lead Time for Changes</strong> Time from commit to production deploy Time between PR merge and successful apply
<strong>Change Failure Rate</strong> % of deploys that cause incidents <code>tofu apply</code> failures + rollback events / total applies
<strong>MTTR</strong> Time to recover from failure Time between alert and successful recovery apply
Capture via: Woodpecker build metadata → Prometheus push gateway or custom exporter → Grafana dashboard.

Phased Rollout

  • Phase 1: tofu fmt + tofu validate on PR (no state access needed)
  • Phase 2: tofu plan on PR (needs kubeconfig + secrets)
  • Phase 3: tofu apply on merge (needs approval gate)
  • Phase 4: DORA metrics dashboard

Decision Needed

Where should pal-e-platform's Woodpecker pipeline live? Options:
  • Mirror to Forgejo — pal-e-platform already lives on GitHub. Mirror it to Forgejo, run CI there. Consistent with all other repos.
  • GitHub webhook to Woodpecker — Keep repo on GitHub, trigger Woodpecker directly. More complex networking.
  • Stay on GitHub — Use GitHub Actions for TF validation. Defeats the self-hosted principle.
Recommendation: Mirror to Forgejo. All CI runs on Woodpecker. GitHub is disaster recovery only.