pal-e-docs

Project: Production Pipeline

project-production-pipeline Project Page

ci-cd active infrastructure
Project

Vision

Every merge to main deploys to production automatically. Zero manual steps. The production pipeline is the system that makes this happen, spanning three infrastructure repos that each own a layer of the deploy flow.
This project is the operational home for that system. It documents how the repos correlate, owns the kanban for pipeline fixes, and serves as the reference for onboarding any new service to production.

User Stories

  • merge-to-deploy: As a developer, when I merge a PR to main, my code reaches production automatically with no manual steps.
  • service-onboarding: As a developer with a local app, I can follow the production-pipeline docs to get it running in prod.
  • pipeline-reliability: The pipeline is reliable across all services. Failures are diagnosable and recoverable without deep infrastructure knowledge.

Architecture

The pipeline has five links. Each must work for every service. A break in any link means code doesn't reach production.
Link Component Owner Repo What It Does
1 Woodpecker CI App repo (<code>.woodpecker.yaml</code>) Builds container image on merge, pushes to Harbor with SHA tag
2 Harbor pal-e-services (Terraform provisions projects + robots) Stores images, provides pull credentials to cluster
3 ArgoCD Image Updater pal-e-services (Helm release + annotations on ArgoCD apps) Polls Harbor for new SHA tags, writes updated tag back to deploy repo via git
4 ArgoCD + CMP pal-e-services (Helm) + pal-e-deployments (manifests) Renders kustomize manifests (including SOPS decryption via CMP sidecar), syncs to cluster
5 Tailscale Funnel pal-e-services (Terraform ingress) Routes external HTTPS traffic to the service pod

Three-Repo Model

Repo Layer What It Owns
<strong>pal-e-platform</strong> Cluster foundation k3s cluster, CNI, storage, Harbor, network policies. The ground floor everything builds on.
<strong>pal-e-services</strong> Service orchestration ArgoCD + Image Updater (Helm), per-service provisioning via <code>var.services</code> in Terraform: Harbor projects, robot accounts, namespaces, pull secrets, ArgoCD apps, funnel ingresses.
<strong>pal-e-deployments</strong> Kubernetes manifests Kustomize bases + per-service overlays. The source ArgoCD watches. Image Updater writes tag updates here via git.

Secrets Strategy: SOPS

Production secrets are SOPS-encrypted in git (*.enc.yaml) within kustomize overlays in pal-e-deployments. ArgoCD decrypts them at render time via the kustomize-sops CMP sidecar on the repo-server. Age key stored in argocd/sops-age-key secret.
This is the standard for all services. The service-onboarding-sop will be updated to reflect this approach (replacing the previous manual kubectl create secret generic guidance).

Deploy Flow

Service Onboarding Quick-Ref

You have a local app (Rails, FastAPI, Node, etc.) and want it in production. Here's what happens in each repo:
Step Repo What You Do
1 App repo Add <code>Dockerfile</code> and <code>.woodpecker.yaml</code> (CI pipeline)
2 pal-e-services Add entry to <code>var.services</code> in <code>k3s.tfvars</code>, <code>tofu apply</code>
3 pal-e-deployments Create <code>overlays/{service}/prod/</code> with kustomization, deployment-patch, SOPS-encrypted secrets
4 pal-e-platform Add namespace to network policies if service needs Postgres/MinIO/Keycloak
5 Woodpecker UI Activate repo, add Harbor secrets via MCP
6 App repo Push to main -- pipeline builds, image updater detects, ArgoCD deploys
Full procedure: Service Onboarding SOP

Status

2026-05-24: CMP SOPS discovery fix merged (PR #98). Added cmp_plugin opt-in to var.services — 5 services now use kustomize-sops CMP for rendering. Image Updater write-back fixed in prior session (switched to git:repocreds + kubernetes mode). Pending: tofu apply to deploy CMP changes, then E2E verification.

Board

See board-production-pipeline

Repos