pal-e-docs

Validation: Migrate westside-admin harbor-creds SOPS to terraform (#143)

validation-143-2026-05-03 Doc

validation pass
Project

Verdict: PASS

Milestone significance: This is the FINAL gate of the 5-day westside-admin bootstrap arc. With this PASS, milestone M1 (first admin login operationally possible) is officially achieved — the only remaining step is the manual Keycloak admin-console click. The deploy chain is healthy end-to-end for the first time since 2026-04-28.

Ticket

forgejo_admin/pal-e-deployments#143 — Migrate westside-admin harbor-creds from SOPS-overlay to terraform-managed (Option A, first migration).
Shipped via PR #145 (squash-merged 2026-05-03, commit a6a99136597d044e619b26b6850bb7be74b24979) plus a coordinated targeted tofu apply against pal-e-services/terraform.

Environment

  • Cluster: prod k3s (single-node, archbox)
  • Namespace: westside-admin
  • External URL: https://westside-admin.tail5b443a.ts.net (Tailscale funnel)
  • ArgoCD app: westside-admin (synced to commit a6a9913)
  • Harbor robot: robot$westside-admin+westside-admin-pull (id=384, project-scoped to westside-admin id=38)
  • Validation run: 2026-05-03 by Dottie

Tier

Per sop-validation, ticket has labels arch:harbor + touches kustomize + coordinated terraform apply. Executed Tier 3 (prod) end-to-end — the live cluster IS the validation surface for an infra-fix bug. Tier 1 (local terraform plan) was rolled in as the no-drift check.

Checks

# Acceptance Criterion How Verified Result Evidence
1 <code>overlays/westside-admin/prod/kustomization.yaml</code> no longer references <code>harbor-creds.enc.yaml</code> in <code>resources:</code> Forgejo API GET on <code>kustomization.yaml@main</code> PASS <code>resources:</code> now lists only <code>namespace.yaml</code>, <code>../../../bases/standard</code>, <code>ingress.yaml</code>, <code>westside-admin-secrets.enc.yaml</code> — no harbor-creds reference
2 <code>overlays/westside-admin/prod/harbor-creds.enc.yaml</code> deleted from repo Forgejo API GET on the file path at <code>ref=main</code> PASS <code>HTTP 404</code> (file does not exist on main)
3 Harbor pull robot exists for westside-admin <code>curl -sk -u admin:$HARBOR_ADMIN_PASSWORD https://harbor.tail5b443a.ts.net/api/v2.0/robots/384</code> PASS Returns <code>name=robot$westside-admin+westside-admin-pull</code>, <code>kind=project</code>, <code>namespace=westside-admin</code>, permissions <code>read artifact</code> + <code>pull repository</code>, <code>disable=false</code>, no expiry
4 <code>kubectl get secret harbor-creds -n westside-admin</code> dockerconfigjson parses cleanly with valid creds <code>kubectl get secret harbor-creds -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | python3 -c json.tool</code> PASS Parses as valid JSON. <code>auths."harbor.tail5b443a.ts.net"</code> contains <code>username=robot$westside-admin+westside-admin-pull</code>, <code>password</code> length=32 (real, not placeholder), <code>auth</code> length=100 (base64-encoded <code>username:password</code>)
5 <code>kubectl get deployment westside-admin -n westside-admin</code> shows 1/1 ready <code>kubectl get deployment westside-admin -n westside-admin -o wide</code> PASS <code>READY=1/1, UP-TO-DATE=1, AVAILABLE=1</code>, image=<code>harbor.tail5b443a.ts.net/westside-admin/app:803e9439...</code>
6 Pod Running, no <code>CreateContainerError</code> or <code>ImagePullBackOff</code> on current pod <code>kubectl get pods</code> + <code>kubectl get events --sort-by=.lastTimestamp</code> PASS Pod <code>westside-admin-5c96ffff7f-rhz84</code> Status=<code>Running</code>, Ready=<code>true</code>, Restarts=<code>0</code>. Latest events show <code>Pulled successfully in 1.522s</code> — the pre-fix events showing <code>illegal base64</code>/<code>FailedToRetrieveImagePullSecret</code> are on superseded ReplicaSets, not current pod
7 <code>/health</code> endpoint responds 200 inside cluster <code>kubectl exec ... -- wget -qO- http://10.42.0.237:3000/health</code> PASS Returns <code>{"status":"ok"}</code>. Note: Alpine wget on <code>localhost</code> hits IPv6 (::1) where node binds IPv4-only — probe via Pod IP works. Kubelet readiness probe uses Pod IP and is passing every 10s (Ready=true confirms)
8 External funnel <code>https://westside-admin.tail5b443a.ts.net</code> returns app (not 502) <code>curl -skI https://westside-admin.tail5b443a.ts.net/</code> PASS <code>HTTP 200</code> in 11.7ms (was 502 since 2026-04-28)
9 No-drift: targeted <code>tofu plan</code> after apply returns 0 changes for the harbor resources <code>cd ~/pal-e-services/terraform &amp;&amp; tofu plan -var-file=k3s.tfvars -target='harbor_robot_account.service_pull["westside-admin"]' -target='kubernetes_secret_v1.harbor_creds["westside-admin"]' -lock=false</code> PASS <code>Plan: 0 to add, 1 to change, 0 to destroy</code>. The 1 change is <code>kubernetes_namespace_v1.service["westside-admin"]</code> (transitive target dep, removing a stray ArgoCD label — pre-existing namespace drift unrelated to harbor migration). The two TARGETED resources (<code>harbor_robot_account</code> + <code>kubernetes_secret_v1.harbor_creds</code>) plan as zero-change — AC condition met
10 ArgoCD sync state Synced (not OutOfSync, not Drift) <code>kubectl get application westside-admin -n argocd -o jsonpath='{.status.sync.status}{" "}{.status.health.status}{" "}{.status.sync.revision}'</code> PASS <code>sync=Synced health=Healthy revision=a6a99136597d044e619b26b6850bb7be74b24979</code> (matches PR #145 squash-merge commit)

Verdict

PASS — all 10 acceptance criteria green with command-output evidence. Move ticket #143 to done.

Regression Check

  • Targeted tofu plan shows the harbor migration introduced no drift on adjacent service harbor_robot/secret resources. Other 13 services still on SOPS-overlay placeholder pattern remain unchanged (per the narrow Option A scope of #143 — their migration is parent ticket #144 follow-up work).
  • ArgoCD westside-admin app health=Healthy confirms no other sync issues introduced by the overlay deletion (kustomize still builds, ingress + namespace + secrets still applied).
  • External funnel responsive (HTTP 200, ~12ms) — ingress + Tailscale funnel still wired correctly. SvelteKit adapter-node serving the app.
  • Pod restart count = 0 on the new pod, no crash-loop. Image pull worked first try with the new credentials.

Discovered Issues

Two incidental observations — neither blocks the verdict:
  • notion-mcp-remote scope creep in same tofu apply. The targeted apply incidentally created harbor_project.service["notion-mcp-remote"] + kubernetes_namespace_v1.service["notion-mcp-remote"] because the service was in var.services but never previously applied. This is benign completion of state drift, not damage. Already tracked context-only (no follow-up ticket needed unless notion-mcp-remote owner objects).
  • Pre-existing namespace label drift on westside-admin namespace. The targeted plan still shows kubernetes_namespace_v1.service["westside-admin"] wants to remove a stray argocd.argoproj.io/instance label. This is a separate pre-existing condition (ArgoCD added the label out-of-band when it adopted the namespace). Cosmetic, not functional. Suggest tracking under a small "argocd-managed-by-label cleanup" hygiene ticket if desired, but NOT a blocker for #143.

Implication for Board Item 1117

Board item 1117 (westside-admin#12 — "Validate step fails: build/index.html doesn't exist for adapter-node") was held in qa pending end-to-end deploy-chain proof. With this PASS, the deploy chain is provably healthy: CI built the image (803e9439...) — that exact tag is now running 1/1 in cluster — the validate-step fix is implicitly verified. Recommend moving #1117 to done alongside #1120 based on this same evidence.

Bootstrap Arc Recap

Three layered blockers, fixed in sequence over 5 days, each unmasking the next:
  • Layer 1 — CI validate step couldn't find build/index.html (adapter-node, not adapter-static). Fixed by westside-admin#13. Surfaced 40-zeros image tag.
  • Layer 2 — PSA admission rejected pod (no securityContext). Fixed by pal-e-deployments#141.
  • Layer 3 — harbor-creds Secret was a SOPS-managed PLACEHOLDER fighting terraform writes. Fixed by THIS ticket (#143/#145) — SOPS overlay deleted, terraform owns it.
The arc demonstrates the value of strict validation gates: each layer was only visible once the previous layer was fixed. Skipping validation would have left this Secret in placeholder state indefinitely.
Parent pal-e-deployments#144 (platform-wide harbor-creds SOPS→terraform migration) now has a working template — the remaining 13 service migrations can follow the same pattern.