pal-e-docs
Project not found.

ArgoCD + Image Updater Deployment Pattern

arch-argocd Doc

architecture argocd platform
Project

Three-Repo Model

Repo Layer Responsibility
<code>pal-e-platform</code> Cluster bootstrap k3s foundation: Tailscale operator, CNPG operator, monitoring stack, Forgejo, Woodpecker CI, Harbor, MinIO, Keycloak. Deploys via OpenTofu with 5 providers and 12 modules. Owns default-deny network policies per namespace.
<code>pal-e-services</code> Application IaC OpenTofu over the running cluster: provisions ArgoCD (Helm), Image Updater, per-service namespaces, Harbor projects, robot accounts, ArgoCD Application CRs, Tailscale funnels, CNPG databases, Keycloak realms/clients.
<code>pal-e-deployments</code> Kubernetes manifests Kustomize overlays that ArgoCD syncs to the cluster. Bases, prod overlays, dev overlays, SOPS-encrypted secrets.
Individual source repos (e.g., basketball-api, paldocs) contain application code, Dockerfiles, and Woodpecker CI pipelines. They feed into the pipeline but are not part of the three-repo model.

The var.services Pattern

A single map in ~/pal-e-services/terraform/k3s.tfvars drives everything. services.tf iterates with for_each and creates up to 7 resources per entry:
Resource Purpose
<code>harbor_project</code> Container image project in Harbor
<code>harbor_robot_account</code> (CI) Push/pull robot for Woodpecker CI
<code>harbor_robot_account</code> (pull) Pull-only robot for image pull secrets
<code>kubernetes_namespace_v1</code> Dedicated namespace
<code>kubernetes_secret_v1</code> <code>harbor-creds</code> pull secret in the namespace
<code>argocd_application</code> ArgoCD app pointing at the overlay or service repo
<code>kubernetes_ingress_v1</code> Tailscale funnel (conditional on <code>funnel = true</code>)
Key fields: forgejo_repo, image_repo, port, funnel. Optional overrides: source_repo / source_path (redirect ArgoCD to pal-e-deployments overlays), cmp_plugin (enable SOPS CMP), target_revision (branch selection).

CI → Registry → Deploy Pipeline

Write-Back

Image Updater uses git write-back: it commits the new image tag into a .argocd-source-* file or kustomization.yaml. The Woodpecker pipeline excludes k8s/.argocd-source-* paths to prevent infinite build loops from write-back commits.

Two Source Patterns

  • Simple services (no SOPS): ArgoCD watches k8s/ in the service's own Forgejo repo. Image Updater writes back to that same repo.
  • Services with secrets: ArgoCD watches overlays/<service>/prod/ in pal-e-deployments. Set source_repo, source_path, and cmp_plugin = "kustomize-sops" in var.services.

Why Image Updater Over CI-Driven Tag Updates

A simpler alternative is having each CI pipeline update the manifests repo directly after pushing an image. Image Updater exists because that approach breaks down at scale:
  • Race condition prevention — When multiple services push images simultaneously, Image Updater serializes the tag updates. Without it, N CI pipelines race to commit to the same manifests repo, causing merge conflicts and lost updates.
  • Ordering guarantees — Image Updater picks the newest build tag by timestamp, not whichever CI pipeline committed last. A slow build that finishes after a fast one doesn't accidentally roll back the fast one's deploy.
  • CI/deploy decoupling — CI pipelines only need push access to Harbor. They don't need credentials to the deployment repo or knowledge of the manifest structure. Adding a new service doesn't require wiring up deployment repo access in its CI pipeline.
  • Single coordination point — One component owns the "what version runs" decision across all services, rather than distributing that responsibility across every CI pipeline.
For a single service, CI-driven updates are simpler. For a platform running 10+ services with independent build pipelines, Image Updater is the coordination layer that prevents the N-pipeline problem.

SOPS CMP Sidecar

The kustomize-sops Config Management Plugin runs as a sidecar on the argocd-repo-server pod. It activates via auto-discovery when a source directory contains both kustomization.yaml and **/*.enc.yaml files.
Render sequence:
  • Find *.enc.yaml files
  • sops --decrypt --in-place each one (using an AGE key mounted from sops-age-key Secret)
  • kustomize build .
  • Return decrypted manifests to ArgoCD for apply
Critical constraint: The ArgoCD Application CR must not have spec.source.kustomize set. Setting it causes ArgoCD to use its built-in kustomize renderer, bypassing the CMP sidecar entirely. Encrypted YAML is applied literally and fails with ENC[AES256_GCM] not found.

Credential Flow

  • forgejo_argocd_token (from k3s.tfvars) → git-creds Secret in argocd namespace → Image Updater uses it to push write-back commits
  • sops-age-private-key (from k3s.tfvars) → sops-age-key Secret in argocd namespace → mounted into CMP sidecar container

Overlay Structure (pal-e-deployments)

Bases

bases/standard/ provides Deployment, Service, ServiceMonitor, NetworkPolicy, and HPA templates. Default port 8000, single replica, conservative resource limits, liveness/readiness on /healthz.

Prod Overlays

overlays/<service>/prod/ references bases/standard, adds JSON patches to rename resources, a deployment-patch.yaml for env vars/secrets/resources, and an images: block with the Harbor image path and commit SHA tag.

Dev Overlays

overlays/<service>/dev/ are standalone (no base reference). Applied manually via kubectl apply -k. ArgoCD does not manage them. Two patterns: host-mounted source code, or nginx proxy to host dev server.

CI Validation

Woodpecker CI on pal-e-deployments PRs runs kubectl kustomize and kubectl apply --dry-run=server for every prod overlay. SOPS-encrypted resources are filtered out (CMP handles decryption at sync time).

Dev/Prod Isolation

Namespace-based. Add a second entry to var.services (e.g., basketball-api-dev) pointing to the same Forgejo repo but a separate Harbor project. Each gets its own namespace, secrets, and funnel URL (basketball-api.tail5b443a.ts.net vs basketball-api-dev.tail5b443a.ts.net). Optional target_revision enables branch-based promotion.

Custom Domain Routing

Services on *.tail5b443a.ts.net get automatic TLS via Tailscale funnels (managed by pal-e-platform's networking module). For non-Tailscale domains (e.g., intelligentstaffingsystems.ai), a Hetzner edge VPS runs Caddy, terminates TLS, and proxies through the Tailscale mesh to the cluster. The edge VPS is provisioned by pal-e-platform's hetzner-edge module.

Network Policies

pal-e-platform/terraform/network-policies.tf defines default-deny ingress for every managed namespace. Each policy allows ingress only from explicitly listed namespaces; egress is unrestricted. The ArgoCD namespace is excluded because pal-e-platform does not manage it -- ArgoCD is deployed by pal-e-services via Helm.

Onboarding a New Service

  • Add entry to services in k3s.tfvars
  • tofu apply -var-file=k3s.tfvars (creates namespace, Harbor project, robots, ArgoCD app, funnel)
  • Create service repo on Forgejo with Dockerfile, .woodpecker.yaml, and k8s/ manifests
  • If SOPS secrets needed: create overlay in pal-e-deployments/overlays/<service>/prod/
  • Configure Woodpecker secrets (harbor_username, harbor_password from tofu output)
  • Push to main -- pipeline fires, image lands in Harbor, Image Updater writes back, ArgoCD syncs
Removal: delete the entry from k3s.tfvars and tofu apply. All 7 resources are destroyed.