Phase: Kustomize Service Bases

phase-pal-e-platform-kustomize Phase

phase

Project

Plan: Kustomize Service Deployment Bases

Goal: Centralize service deployment configuration into Kustomize bases and overlays. Every service gets platform conventions (HPA, ServiceMonitor, resource limits) by default. One PR to the base = every service inherits the change.

Owner: Dev agent (per repo)

Repo: forgejo_admin/pal-e-deployments (primary), forgejo_admin/pal-e-services, service repos

Depends on: None (observability stack complete, SOPS CMP sidecar already deployed)

Scope

Four platform repos first, then propagate to app repos one at a time:

pal-e-deployments — kustomize bases + overlays (the new repo, already renamed)
pal-e-platform — where this plan lives
pal-e-services — Terraform service onboarding (ArgoCD source paths, Image Updater annotations)
pal-e-agency — convention/SOP that crystallizes from the first migration

Ordering (Lucas's direction, 2026-03-14)

The convention can't be defined in a vacuum. It emerges from doing the first migration.

Migrate pal-e-docs — build the kustomize bases in pal-e-deployments as you go. The base templates and overlay pattern emerge from real work.
Document the convention — the SOP in pal-e-agency captures what actually worked. Not theory, practice.
Update pal-e-services — Terraform service onboarding generates overlays pointing at the bases.
Propagate — remaining repos one at a time, following the convention.

What's Changed Since Original Scoping (2026-02-27)

deployments repo renamed to pal-e-deployments (done)
pal-e-docs migrated from SQLite/Litestream to CNPG Postgres + embedding-worker. Overlay needs: deployment, service, servicemonitor, embedding-worker deployment. No more litestream sidecar/configmap/PVC.
SOPS CMP sidecar already deployed on ArgoCD (PR #9 on pal-e-services, 2026-03-14). Age key operational.
pal-e-agency added as touchpoint — convention-kustomize-overlay SOP emerges from Phase 7a work.
basketball-api-dev namespace decommissioned (Phase 14b). No dev overlay needed currently.

Decisions Made

Decision	Rationale
Centralized bases in pal-e-deployments	Platform-wide changes = one PR. Consistency enforced, not documented-and-hoped.
Terraform for_each stays for infrastructure envelope	Harbor, namespaces, secrets, funnels — cross-provider orchestration needs state. Only ArgoCD source path changes.
HPA in base (default minReplicas: 1, maxReplicas: 1)	Pattern established, services opt in via overlay patch.
Image Updater kustomization write-back mode	Updates images: in kustomization.yaml directly. Avoids merge conflicts in centralized repo.
Convention emerges from first migration, not upfront	Lucas directive 2026-03-14. Can't define the SOP in a vacuum.

Phases

Phase 7a: Base Structure + Migrate pal-e-docs (COMPLETED)

Goal: Kustomize base exists. pal-e-docs runs from centralized overlay. Image Updater writes back to overlay. Pattern proven end-to-end.

Create base structure in pal-e-deployments:

bases/
  standard/
    kustomization.yaml
    deployment.yaml
    service.yaml
    hpa.yaml
    servicemonitor.yaml

Create pal-e-docs overlay (Postgres-based, with embedding-worker):

overlays/
  pal-e-docs/
    prod/
      kustomization.yaml
      deployment-patch.yaml
      embedding-worker.yaml

Update ArgoCD Application source in pal-e-services Terraform
Verify: deploy from overlay, Image Updater kustomization write-back, full push-build-deploy loop

Phase 7b: Document Convention + SOP

Goal: Convention note in pal-e-agency captures what emerged from 7a. Service onboarding SOP updated.

Create convention-kustomize-overlay in pal-e-docs (pal-e-agency project)
Update service-onboarding-sop to reference kustomize workflow
Update SERVICE_ONBOARDING.md in pal-e-services

Phase 7c: Update pal-e-services Terraform (COMPLETED — absorbed into 7a)

Status: COMPLETED. Absorbed into Phase 7a. The source_repo + source_path fields and the conditional write-back-target: kustomization annotation were implemented in pal-e-services PR #11 as part of the pal-e-docs migration. The terraform coalesce() pattern means every service gets the kustomize write-back target automatically when source_repo is set in tfvars.

Add source_repo + source_path fields to var.services
Add write-back-target: kustomization annotation to all ArgoCD Applications

Phase 7d: Propagate to remaining services (COMPLETED — 4/4 migrated)

Goal: All services on centralized overlays. k8s/ directories retained as fallback.

Progress (2026-03-15):
- basketball-api — MIGRATED. Overlay in overlays/basketball-api/prod/. Includes postgres sidecar, photo PVC, Stripe + Keycloak env vars. Found pre-existing migration bug (sa.Enum(create_type=False) doesn't prevent DDL events — fixed with postgresql.ENUM).
- westsidekingsandqueens — MIGRATED. Overlay in overlays/westsidekingsandqueens/prod/. Port 3000 override, SOPS encrypted auth secret, Tailscale funnel Ingress. Also fixed CI root cause: $CI_COMMIT_SHA not expanding (needs curly braces ${CI_COMMIT_SHA}). Security fix (access token leak) deployed.
- platform-validation — MIGRATED (2026-03-15). Overlay in overlays/platform-validation/prod/. Simplest service: port 8080, 64Mi memory, no secrets/PVC/DB. Learned: kustomize strategic merge appends ports by containerPort key — use JSON patch (op: replace) to override port arrays.
- pal-e-app — MIGRATED (2026-03-15). Overlay in overlays/pal-e-app/prod/. Port 3000, SOPS encrypted auth secrets (pal-e-auth-secrets.enc.yaml), Keycloak + pal-e-docs env vars, health check on / not /healthz.

All 6 ArgoCD apps now read from pal-e-deployments. Zero services remaining on app repo k8s/ directories.

basketball-api, westside-app, pal-e-app, platform-validation — one at a time
Each follows the convention from 7b

Verification

[ ] 7a: pal-e-docs deploys from centralized overlay. Image Updater kustomization write-back works. Full CI/CD loop verified.
[ ] 7b: Convention note exists. Service onboarding SOP updated.
[ ] 7c: pal-e-services Terraform generates correct ArgoCD source paths.
[ ] 7d: All services on centralized overlays. No more k8s/ in service repos.

plan-pal-e-platform — parent plan
phase-pal-e-platform-network-security — Phase 8, will add NetworkPolicy to the base
phase-pal-e-platform-env-isolation — Phase 9, dev/prod separation
service-onboarding-sop — rewritten in Phase 7b
Forgejo issue: pal-e-services #1 (Kustomize Phase 1)