pal-e-docs

Validation: pal-e-deployments prod/dev kustomize overlays for ISS

validation-78-2026-07-17 Doc

validation partial
Project

Verdict: PARTIAL

Ticket

intelligentstaffingsystems#78 — Create prod and dev kustomize overlays for ISS in pal-e-deployments. Merged via pal-e-deployments#234.

Environment

Production cluster, namespace intelligentstaffingsystems, ArgoCD application intelligentstaffingsystems sourcing from overlays/intelligentstaffingsystems/prod in pal-e-deployments. Public URL: https://intelligentstaffingsystems.ai.

Checks

# Criterion How Verified Result Evidence
1 ArgoCD syncs the prod overlay <code>kubectl get application -n argocd intelligentstaffingsystems -o jsonpath='{.status.sync.status}'</code> PASS Returns <code>Synced</code>. Source path: <code>overlays/intelligentstaffingsystems/prod</code> from pal-e-deployments.
2 ArgoCD application is healthy <code>kubectl get application -n argocd intelligentstaffingsystems -o jsonpath='{.status.health.status}'</code> FAIL Returns <code>Degraded</code>. Deployment resource is degraded; Service is Healthy; Secret and ServiceMonitor are synced.
3 Pods running new image, not crash-looping <code>kubectl get pods -n intelligentstaffingsystems</code> FAIL Two pods exist: <code>intelligentstaffingsystems-6d6896dd95-ls9k4</code> (1/1 Running, 0 restarts, 11h old) and <code>intelligentstaffingsystems-8474c67bc8-nxphp</code> (0/1 Init:CrashLoopBackOff, 13 restarts). Both use image <code>harbor.tail5b443a.ts.net/intelligentstaffingsystems/app:93b9a79b8a88cbb07a7f8fda1b6d85803580cbbb</code>.
4 Secret values are correct (not placeholders) <code>kubectl get secret -n intelligentstaffingsystems intelligentstaffingsystems-secrets</code> (decoded) FAIL Four keys contain placeholder values: <code>POSTGRES_HOST=PLACEHOLDER_POSTGRES_HOST</code>, <code>POSTGRES_USER=PLACEHOLDER_POSTGRES_USER</code>, <code>APP_URL=PLACEHOLDER_APP_URL</code>, <code>POSTMARK_API_TOKEN=PLACEHOLDER_POSTMARK_API_TOKEN</code>. Keycloak keys are correctly populated.
5 Production site accessible <code>curl -sk -o /dev/null -w "%{http_code}" https://intelligentstaffingsystems.ai/</code> PASS Returns HTTP 200. Site loads correctly from the old pod that cached correct secret values at startup.
6 Dev overlay deployed via ArgoCD <code>kubectl get applications -n argocd</code> filtered for ISS FAIL Only <code>intelligentstaffingsystems</code> (prod) exists. No dev ArgoCD Application found. No <code>intelligentstaffingsystems-dev</code> namespace exists either.

Root Cause Analysis

The kustomize overlay in PR #234 includes a Secret manifest with placeholder values. When ArgoCD synced, it overwrote the previously-manually-populated secret with the placeholder values from the overlay. The old pod (replicaset 6d6896dd95) survives because Kubernetes does not restart running pods when referenced secrets change (for secretKeyRef env vars). The new pod (replicaset 8474c67bc8) picks up the placeholder values at creation time and the migrate init container fails with ActiveRecord::DatabaseConnectionError: There is an issue connecting with your hostname: PLACEHOLDER_POSTGRES_HOST.
The deployment shows ProgressDeadlineExceeded on the new replicaset.

Regression Check

The production site (https://intelligentstaffingsystems.ai) remains accessible via the old pod. However, this is fragile: any pod restart, scaling event, or deployment rollout will cause all pods to pick up the placeholder secrets and crash. The next ISS image push from Woodpecker CI will trigger a rollout that replaces the old working pod, taking the site down.

Discovered Issues

  • Placeholder secrets in kustomize overlay: The Secret manifest in overlays/intelligentstaffingsystems/prod contains placeholder values for POSTGRES_HOST, POSTGRES_USER, APP_URL, and POSTMARK_API_TOKEN. These must be replaced with actual values, either via SealedSecrets, manual kubectl apply, or removing the Secret from the overlay and managing it out-of-band.
  • No dev ArgoCD Application: The dev overlay may exist in the pal-e-deployments repo, but no ArgoCD Application deploys it. A dev ArgoCD app and namespace need to be created.
  • Imminent production risk: The next CI image push will trigger a full rollout, replacing the old working pod and causing a production outage due to the placeholder secrets.