pal-e-docs

Validation: Replace ISS prod auto-deploy with manual Woodpecker promotion pipeline

validation-200-2026-07-26 Doc

validation fail
Project

Verdict: FAIL

Ticket

ldraney/pal-e-services#200 — Replace ISS prod auto-deploy with manual Woodpecker promotion pipeline. Board item #1904 on board-iss. Closed by PR #201 (merged 2026-07-26). Companion PR: ldraney/intelligentstaffingsystems#109 (promote-to-prod Woodpecker step, merged).

Environment

Prod cluster (k3s on archbox). Namespace: intelligentstaffingsystems. ArgoCD app: intelligentstaffingsystems. Public URL: https://intelligentstaffingsystems.ai. Tailscale URL: https://intelligentstaffingsystems.tail5b443a.ts.net.

Checks

# Criterion How Verified Result Evidence
1 Merging to main does NOT auto-deploy to prod (image updater annotations removed from prod ArgoCD app) <code>kubectl get application intelligentstaffingsystems -n argocd -o jsonpath='{.metadata.annotations}'</code> FAIL 6 image-updater annotations still present: <code>git-branch</code>, <code>image-list</code>, <code>allow-tags</code>, <code>update-strategy</code>, <code>write-back-method</code>, <code>write-back-target</code>. Root cause: PR #201 only changed <code>k3s.tfvars.example</code> (the example file). The live <code>k3s.tfvars</code> (SOPS-encrypted) was NOT updated with <code>image_updater = false</code>. Pipeline #270 ran <code>tofu apply</code> but applied zero ISS-related changes (only mdview and gcal-scheduler drift).
2 Merging to main still auto-deploys to staging via image updater (staging ArgoCD app unchanged) <code>kubectl get application intelligentstaffingsystems-staging -n argocd</code> FAIL ArgoCD app <code>intelligentstaffingsystems-staging</code> does not exist on the cluster. Error: <code>NotFound</code>. Staging environment from PR #196 was never deployed.
3 A promote-to-prod step exists in the ISS Woodpecker pipeline, gated to event: manual Read <code>.woodpecker.yaml</code> in intelligentstaffingsystems repo PASS Step <code>promote-to-prod</code> present with <code>when: - event: manual</code>, uses <code>alpine/git:latest</code>, depends on <code>build-and-push</code>. Delivered by commit <code>5b227d9</code> (PR #109).
4 The promote step sets DEPLOY_REPO=ldraney/pal-e-deployments Read <code>.woodpecker.yaml</code> environment block PASS <code>DEPLOY_REPO: "ldraney/pal-e-deployments"</code> is set in the step environment, overriding the default <code>forgejo_admin/pal-e-deployments</code>.
5 Manually triggering the Woodpecker pipeline updates prod kustomization.yaml newTag Checked <code>mcp__woodpecker__list_pipelines</code> for manual events on ISS repo BLOCKED No manual pipeline triggers have ever been run on the ISS repo. Cannot verify end-to-end promotion flow without triggering it.
6 ArgoCD picks up the kustomization change and deploys the specified image to prod N/A BLOCKED Depends on AC #5; cannot verify without a manual trigger.
7 Prod remains Synced and Healthy in ArgoCD after the change <code>kubectl get application intelligentstaffingsystems -n argocd</code> PASS Status: Synced, Health: Healthy. Pod <code>intelligentstaffingsystems-69c4f699d6-vp89s</code> Running with 0 restarts, image tag <code>b46c433...</code>.
8 tofu plan is clean after apply (no image updater annotation drift) Reviewed Woodpecker pipeline #270 apply step logs FAIL Apply ran successfully but applied zero ISS changes. The plan/apply only touched mdview namespace labels and gcal-scheduler harbor creds. Live <code>k3s.tfvars</code> does not contain <code>image_updater = false</code> for ISS, so no annotation removal was attempted.
9 kubectl shows no image-updater annotations on prod ArgoCD app <code>kubectl get application intelligentstaffingsystems -n argocd -o yaml | grep image-updater</code> FAIL Returns 6 lines of image-updater annotations. Auto-deploy is still active for prod.

Regression Check

Prod pod is healthy (Running, 0 restarts). Public URL https://intelligentstaffingsystems.ai returns HTTP 200. ArgoCD app is Synced/Healthy. No regressions introduced -- the terraform change was simply never applied to the ISS service.

Discovered Issues

  • Live k3s.tfvars not updated. PR #201 only changed k3s.tfvars.example. The live SOPS-encrypted k3s.tfvars still has the ISS prod service defaulting to image_updater = true. The PR body explicitly notes this as a manual step: "the live k3s.tfvars also needs image_updater = false added to the intelligentstaffingsystems block." This manual step was not completed.
  • No staging ArgoCD app. intelligentstaffingsystems-staging does not exist on the cluster. PR #196 (staging environment) was merged in pal-e-services but the staging app was never created. AC #2 depends on staging existing.
  • Manual promotion never tested. The promote-to-prod Woodpecker step exists but has never been triggered. End-to-end flow (manual trigger -> kustomization update -> ArgoCD deploy) is unverified.