Phase: CI Pipeline & Team Hardening

Plan: Platform CI Pipeline & Team Hardening

Vision

Projects & Repos Touched

Context

• [x] Both repos use kubernetes state backend with locking (tofu-state namespace)
• [x] Woodpecker CI deployed and proven (builds app container images for 4 services)
• [x] MinIO deployed (backup target for state)
• [x] TF assessment complete — 8 deep-dive notes documenting gaps and recommendations
• [x] Both project pages updated with architecture, decisions, and reference notes
• [x] Salt plan complete — host is managed, secrets encrypted, firewall active
• [ ] No CI pipeline for Terraform itself
• [ ] No automated state backup
• [ ] No secrets management beyond plaintext tfvars
• [ ] No developer onboarding docs for TF work
• [ ] No disaster recovery runbook

Previous Plan

Depends On

Decisions Made

Phases

Phase 1: State Backup CronJob

• Create a CronJob in tofu-state namespace that:
• Deploy CronJob via Terraform in pal-e-platform (new resource in main.tf or future module)
• Create MinIO IAM user + policy scoped to litestream-backups/tf-state/ prefix
• Write restore SOP in pal-e-docs
• Test: manually trigger CronJob, verify backup exists in MinIO, test restore to a temp secret

• Full rebuild sequence: Arch install → clone repo → recover GPG key → Salt bootstrap → Salt apply → tofu apply → verify
• What's recoverable: TF state (from MinIO backup), pal-e-docs data (Litestream/MinIO — same host caveat), Forgejo repos (GitHub mirrors)
• What's NOT recoverable without off-host backups: Harbor images, MinIO data if NVMe fails, any non-mirrored Forgejo repos
• GPG key recovery from physical backup (see todo-gpg-physical-backup)

Phase 2: Mirror Repos to Forgejo + Validation Pipeline

• Create mirror repos on Forgejo:
• Configure Forgejo mirror sync (Settings, Mirror, pull from GitHub on schedule)
• Activate both repos in Woodpecker UI
• Add .woodpecker.yaml to each repo:

  steps:
    validate:
      image: ghcr.io/opentofu/opentofu:1.9
      commands:
        - cd terraform
        - tofu init -backend=false
        - tofu fmt -check -recursive
        - tofu validate
  

• Enable Forgejo branch protection on main for both repos:
• Verify: push a branch, PR shows green/red from Woodpecker. Direct push to main is rejected.

Phase 3: Plan-on-PR Pipeline

• Add Woodpecker secrets for each repo:
• Update .woodpecker.yaml to add plan step:

  steps:
    plan:
      image: ghcr.io/opentofu/opentofu:1.9
      commands:
        - mkdir -p ~/.kube
        - echo "$KUBECONFIG_CONTENT" > ~/.kube/config
        - cd terraform
        - tofu init
        - tofu plan -no-color 2>&1 | tee plan.txt
        - # Post plan.txt as PR comment via Forgejo API
      secrets: [kubeconfig_content, tf_var_...]
      when:
        event: pull_request
  

• Write a small script or use Woodpecker plugin to post plan output as PR comment
• Verify: open PR with a TF change, plan appears as comment, reviewer can assess impact

Phase 4: Apply-on-Merge Pipeline

• Add apply step to .woodpecker.yaml:

  steps:
    apply:
      image: ghcr.io/opentofu/opentofu:1.9
      commands:
        - mkdir -p ~/.kube
        - echo "$KUBECONFIG_CONTENT" > ~/.kube/config
        - cd terraform
        - tofu init
        - tofu apply -auto-approve
      secrets: [kubeconfig_content, tf_var_...]
      when:
        event: push
        branch: main
  

• Add pipeline ordering: pal-e-platform applies before pal-e-services (manual gate or cross-repo trigger)
• Add failure notification: Woodpecker webhook or Forgejo notification on apply failure
• Document the new workflow SOP: "How to make infrastructure changes"
• Agreement: nobody runs tofu apply from their laptop once this is live (break-glass exception with documented procedure)
• Verify: merge a trivial change (e.g., add a comment), confirm apply runs and succeeds

Phase 5: Developer Onboarding & DORA Baseline

• Write developer onboarding SOP in pal-e-docs:
• Create RBAC: per-developer kubeconfig with read-only cluster access (sufficient for tofu plan)
• Establish DORA baseline from pipeline data:
• Optional: scheduled tofu plan (drift detection) that alerts if plan shows unexpected changes
• Update both project pages with pipeline status and onboarding link

Key Files

Verification

• [ ] Phase 1: State backup exists in MinIO for both repos. Restore SOP tested. DR runbook written.
• [ ] Phase 2: Both repos mirrored on Forgejo. PR triggers fmt+validate in Woodpecker. Branch protection prevents direct push to main.
• [ ] Phase 3: PR to either repo gets a tofu plan output posted as comment.
• [ ] Phase 4: Merge to main triggers tofu apply. Trivial change applies successfully.
• [ ] Phase 5: Onboarding SOP written. DORA baseline captured. A second developer could follow the doc and submit a TF change.

Next Plan Seeds

• Drift detection — scheduled tofu plan that alerts on unexpected changes (manual kubectl edits, Helm drift)
• DORA metrics Grafana dashboard — Woodpecker build metadata to Prometheus to Grafana visualization
• SOPS or Vault for secrets — if team grows beyond 2-3 developers, Woodpecker secrets become unwieldy. SOPS + age encrypts tfvars in git.
• State splitting — for larger teams, split pal-e-platform state per module (monitoring, forgejo, harbor) to reduce blast radius and enable parallel applies
• Platform dev cluster — second k3s for testing Helm upgrades and TF changes before prod. Requires modularization (covered by other plan).
• GitHub to Forgejo full migration — move pal-e-platform and pal-e-services source of truth from GitHub to Forgejo. GitHub becomes read-only mirror.
• Off-host MinIO replication — replicate MinIO bucket to external S3 (Backblaze B2, Hetzner Object Storage) so NVMe failure doesn't lose all backups. Critical for a complete DR story.

Related

• plan-2026-02-26-tf-modularize-postgres — companion plan covering modularization and Postgres. Independent but complementary.
• plan-2026-02-25-platform-observability — alerting from Phase 3 of that plan feeds into MTTR measurement in Phase 5 of this plan.
• plan-2026-02-26-salt-host-management (COMPLETE) — DR runbook redistributed from that plan's deferred Phase 4. Salt CI is a future extension of TF CI.
• tf-architecture-assessment-2026-02-26 — the assessment that generated this plan (hub note linking 8 deep-dives).
• tf-pipeline-design — detailed pipeline design note with Woodpecker YAML examples and challenge table.
• tf-rollback-strategy — rollback mechanisms. Phase 4 enables git-revert-based rollback.
• tf-team-readiness — the 7 blockers this plan addresses.
• tf-best-practices-comparison — priority order that informed phase sequencing.
• service-onboarding-sop — the service-level SOP. This plan creates the infrastructure-level equivalent.
• todo-deployment-safety — the incident that motivated this work.
• todo-gpg-physical-backup — GPG backup prerequisite for DR runbook.
• platform-maturity-matrix — maps this plan's phases to enterprise capability targets.