Validation: Replace ISS prod auto-deploy with manual Woodpecker promotion pipeline
Verdict: PARTIAL
Ticket
ldraney/pal-e-services#200 — Disable ISS prod auto-deploy via ArgoCD Image Updater; add manual promote-to-prod Woodpecker pipeline step.
Environment
Production cluster (k3s, archbox node). Namespaces:
intelligentstaffingsystems (prod), intelligentstaffingsystems-staging (staging). ArgoCD namespace: argocd. Terraform workspace: ~/pal-e-services/terraform with k3s.tfvars.Checks
| # | Criterion | How Verified | Result | Evidence |
|---|---|---|---|---|
| 1 | Merging to main does NOT auto-deploy to prod (image updater annotations removed) | <code>kubectl get applications.argoproj.io intelligentstaffingsystems -n argocd -o yaml | grep image-updater</code> | PASS | No output — no image-updater annotations on prod ArgoCD app |
| 2 | Merging to main still auto-deploys to staging | <code>kubectl get applications.argoproj.io intelligentstaffingsystems-staging -n argocd -o yaml | grep image-updater</code> | PASS | Staging app has full image-updater annotations: <code>argocd-image-updater.argoproj.io/image-list</code>, <code>update-strategy: newest-build</code>, <code>write-back-method: git:repocreds</code> |
| 3 | promote-to-prod step exists in Woodpecker pipeline, gated to event: manual | Read <code>.woodpecker.yaml</code> in intelligentstaffingsystems repo; checked pipeline #188 (push event) step list | PASS | Step <code>promote-to-prod</code> present with <code>when: event: manual</code>. Pipeline #188 (push) ran 7 steps; promote-to-prod was NOT among them. |
| 4 | promote step sets DEPLOY_REPO=ldraney/pal-e-deployments | Read <code>.woodpecker.yaml</code> environment block | PASS | <code>DEPLOY_REPO: "ldraney/pal-e-deployments"</code> confirmed in pipeline config |
| 5 | Manual trigger updates prod overlay newTag in pal-e-deployments | Checked Woodpecker manual pipeline history | UNTESTED | No manual pipelines have been triggered (<code>list_pipelines event:manual</code> returned empty). Configuration is correct and follows proven westside-basketball pattern. |
| 6 | ArgoCD picks up kustomization change and deploys to prod | Verified ArgoCD source_path config | UNTESTED | ArgoCD app watches <code>overlays/intelligentstaffingsystems/prod</code> in pal-e-deployments. Pattern proven with other services. Requires AC#5 to trigger first. |
| 7 | Prod remains Synced and Healthy in ArgoCD | <code>kubectl get applications.argoproj.io intelligentstaffingsystems -n argocd</code> | PASS | Status: Synced, Health: Healthy. Pod: 1/1 Running, 0 restarts. |
| 8 | tofu plan is clean (no image updater annotation drift) | <code>tofu plan -var-file=k3s.tfvars -lock=false</code> | PASS | Plan shows 6 changes — all unrelated services (mdview label, gcal-scheduler secret, keycloak smtp sensitivity). Zero ISS-related drift. <code>image_updater = false</code> confirmed in k3s.tfvars line 382. |
| 9 | kubectl grep image-updater returns empty on prod app | <code>kubectl get applications.argoproj.io intelligentstaffingsystems -n argocd -o yaml | grep image-updater</code> | PASS | No output. Prod app metadata.annotations is empty. |
Regression Check
- ISS prod pod running: 1/1, 0 restarts, image
b46c43399b... - ISS staging pod running: 1/1, 0 restarts, Synced/Healthy
- Prod URL (intelligentstaffingsystems.ai) returns HTTP 200
- tofu plan shows no ISS-related resource changes
- Woodpecker pipeline #188 (latest push to main) succeeded — build-and-push completed, promote-to-prod correctly skipped
Discovered Issues
- staging.intelligentstaffingsystems.ai returns connection refused (curl exit 6). The staging ArgoCD app is Synced/Healthy and the pod is running, but the external URL is unreachable. Likely a Tailscale funnel or DNS issue for the staging subdomain. Not a regression from this ticket (staging was just created by #196). Recommend a separate issue to wire the staging funnel.
- End-to-end promote flow untested. The promote-to-prod pipeline step has never been manually triggered. The first real prod deploy via this pipeline will be the true acceptance test for AC #5 and #6. Configuration matches the proven westside-basketball pattern.