pal-e-docs

Production Pipeline

production-pipeline forgejo

Board

Notes

Project Page 1
  • Project: Production Pipeline project-production-pipeline

    Vision

    Every merge to main deploys to production automatically. Zero manual steps. The production pipeline is the system that makes this happen, spanning three infrastructure repos that each own a layer of the deploy flow.

    This project is the operational home for that system. It documents how the repos correlate, owns the kanban for pipeline fixes, and serves as the reference for onboarding any new service to production.

    User Stories

    • merge-to-deploy: As a developer, when I merge a PR to main, my code reaches production automatically with no manual steps.
    • service-onboarding: As a developer with a local app, I can follow the production-pipeline docs to get it running in prod.
    • pipeline-reliability: The pipeline is reliable across all services. Failures are diagnosable and recoverable without deep infrastructure knowledge.

    Architecture

    The pipeline has five links. Each must work for every service. A break in any link means code doesn't reach production.

    Link Component Owner Repo What It Does
    1 Woodpecker CI App repo (.woodpecker.yaml) Builds container image on merge, pushes to Harbor with SHA tag
    2 Harbor pal-e-services (Terraform provisions projects + robots) Stores images, provides pull credentials to cluster
    3 ArgoCD Image Updater pal-e-services (Helm release + annotations on ArgoCD apps) Polls Harbor for new SHA tags, writes updated tag back to deploy repo via git
    4 ArgoCD + CMP pal-e-services (Helm) + pal-e-deployments (manifests) Renders kustomize manifests (including SOPS decryption via CMP sidecar), syncs to cluster
    5 Tailscale Funnel pal-e-services (Terraform ingress) Routes external HTTPS traffic to the service pod

    Three-Repo Model

    Repo Layer What It Owns
    pal-e-platform Cluster foundation k3s cluster, CNI, storage, Harbor, network policies. The ground floor everything builds on.
    pal-e-services Service orchestration ArgoCD + Image Updater (Helm), per-service provisioning via var.services in Terraform: Harbor projects, robot accounts, namespaces, pull secrets, ArgoCD apps, funnel ingresses.
    pal-e-deployments Kubernetes manifests Kustomize bases + per-service overlays. The source ArgoCD watches. Image Updater writes tag updates here via git.

    Secrets Strategy: SOPS

    Production secrets are SOPS-encrypted in git (*.enc.yaml) within kustomize overlays in pal-e-deployments. ArgoCD decrypts them at render time via the kustomize-sops CMP sidecar on the repo-server. Age key stored in argocd/sops-age-key secret.

    This is the standard for all services. The service-onboarding-sop will be updated to reflect this approach (replacing the previous manual kubectl create secret generic guidance).

    Deploy Flow

    Developer merges PR to main (app repo)
  |
  v
Forgejo webhook fires --> Woodpecker CI
  |
  v
Kaniko builds image --> pushes harbor.tail5b443a.ts.net/{project}/app:{commit-sha}
  |
  v
ArgoCD Image Updater (polling every 2min)
  - Detects new SHA tag matching regexp:^[0-9a-f]{7,40}$
  - Picks newest by build date
  - Git write-back: commits updated newTag to pal-e-deployments
  |
  v
ArgoCD (watching pal-e-deployments, auto-sync enabled)
  - Detects kustomization.yaml changed
  - CMP sidecar decrypts *.enc.yaml via SOPS
  - kustomize build renders final manifests
  - Syncs to cluster: rolling update with new image
  |
  v
Tailscale Funnel routes traffic to new pod
  - {service}.tail5b443a.ts.net is live

    Service Onboarding Quick-Ref

    You have a local app (Rails, FastAPI, Node, etc.) and want it in production. Here's what happens in each repo:

    Step Repo What You Do
    1 App repo Add Dockerfile and .woodpecker.yaml (CI pipeline)
    2 pal-e-services Add entry to var.services in k3s.tfvars, tofu apply
    3 pal-e-deployments Create overlays/{service}/prod/ with kustomization, deployment-patch, SOPS-encrypted secrets
    4 pal-e-platform Add namespace to network policies if service needs Postgres/MinIO/Keycloak
    5 Woodpecker UI Activate repo, add Harbor secrets via MCP
    6 App repo Push to main -- pipeline builds, image updater detects, ArgoCD deploys

    Full procedure: Service Onboarding SOP

    Status

    2026-05-24: CMP SOPS discovery fix merged (PR #98). Added cmp_plugin opt-in to var.services — 5 services now use kustomize-sops CMP for rendering. Image Updater write-back fixed in prior session (switched to git:repocreds + kubernetes mode). Pending: tofu apply to deploy CMP changes, then E2E verification.

    Board

    See board-production-pipeline

    Repos

Board 1