pal-e-docs

TODO: Fix Harbor imagePullSecret drift across namespaces

todo-harbor-pull-secret-drift Doc

open todo
Project

TODO: Fix Harbor imagePullSecret drift across namespaces

Problem

The westsidekingsandqueens namespace had a harbor-creds secret using robot account robot$westsidekingsandqueens+westsidekingsandqueens-pull, but the CI pipeline pushes images to Harbor project westside-app/app. The robot is scoped to the wrong Harbor project → 401 Unauthorized on image pull → ImagePullBackOff → deploy silently fails while old pod keeps running.
This is the second time this exact issue has caused a deploy failure. Session 2026-03-14 fixed it with admin creds, but something (likely ArgoCD re-sync or kustomize re-apply) reverted the secret to the wrong robot account.

Impact

3 PRs merged to main (PRs #23, #25, #27), 3 CI pipelines all green, but zero changes deployed. The old pod kept serving stale code. Lucas saw the broken mobile nav and thought nothing had deployed. This is a silent deploy failure — the worst kind.

Root Cause Analysis Needed

  • Why does the secret revert? Is it in a kustomize overlay? ArgoCD managed? Terraform?
  • The harbor-creds secret in the kustomize base or overlay likely has the wrong robot baked in
  • Check pal-e-deployments/overlays/westsidekingsandqueens/ for a SealedSecret or secret generator
  • Check if other namespaces have the same problem (basketball-api, pal-e-docs, etc.)

Fix Requirements (updated 2026-03-17)

Investigation (2026-03-17): The root cause is an architectural gap. harbor-creds is referenced in pal-e-deployments/bases/standard/deployment.yaml (line 20-21) but NO Secret definition exists anywhere — not in kustomize, SOPS, or Terraform. All 11 overlays inherit the reference. The secret is created manually via kubectl, which drifts on ArgoCD sync. Recommended fix: SOPS path. Create harbor-creds.enc.yaml in each overlay, encrypted with the Age key already deployed to ArgoCD (age15ct78fr4scv4vxzj3k6q76wshywzlu0mdc64a624e264dst7zfaq6tjzjr). ArgoCD decrypts at sync time. This makes the secret reproducible, git-tracked, and survives cluster rebuilds. PR to pal-e-deployments repo.
  • Harbor pull secrets must use credentials scoped to the correct Harbor project
  • Secrets must be SOPS-encrypted in the kustomize overlay (not manually applied via kubectl)
  • ArgoCD must not revert manually-fixed secrets on re-sync
  • Consider: one Harbor robot per namespace with correct project scope, OR a single cluster-wide pull secret
  • Add Blackbox probe or alert for ImagePullBackOff events (Phase 10d-4/10d-5 scope)

Temporary Fix Applied

Session 2026-03-15: Replaced harbor-creds in westsidekingsandqueens namespace with Harbor admin creds via kubectl create secret. This is a manual fix and will be reverted on next ArgoCD sync if the overlay has a conflicting secret definition.

Lineage

plan-pal-e-platform — Platform Hardening. This is an operational reliability issue that directly impacts Deployment Frequency and Change Failure Rate.