pal-e-docs

SOP: Deploy Recovery

sop-deploy-recovery Sop

sop active
Project

SOP: Deploy Recovery

Purpose: Teach agents how to diagnose and recover from deployment failures across the ArgoCD + k3s stack. Covers sync failures, pod crashes, image pull errors, CrashLoopBackOff, and the ArgoCD ghost override problem.
Traceability: plan-pal-e-agency → Phase 5 (Error Recovery SOPs)

Failure Modes

Symptom Likely Cause Recovery Steps
ArgoCD shows <strong>OutOfSync</strong> but auto-sync is enabled Manifest in app repo diverged from what ArgoCD last applied. Could be manual kubectl edit (selfHeal reverts these) or a merge that ArgoCD has not detected yet. 1. Check ArgoCD app status: <code>kubectl -n argocd get application NAME -o yaml</code>. 2. Force refresh: <code>kubectl -n argocd patch application NAME --type merge -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"normal"}}}'</code>. 3. If still out of sync, check the app repo <code>k8s/</code> directory for errors.
ArgoCD sync fails with <strong>ComparisonError</strong> Invalid YAML in k8s manifests, or a CRD referenced that does not exist on the cluster. 1. Read the sync error message in ArgoCD UI or <code>kubectl -n argocd get application NAME -o jsonpath='{.status.conditions}'</code>. 2. Validate manifests locally: <code>kubectl apply --dry-run=client -f k8s/</code> (or <code>-k k8s/</code> if kustomize). 3. Fix and push.
Pod in <strong>CrashLoopBackOff</strong> App crashing on startup. Common causes: missing env var, DB connection refused, migration failure, OOM kill. 1. Check logs: <code>kubectl logs -n NAMESPACE POD --previous</code> (the <code>--previous</code> flag gets logs from the crashed container). 2. Check events: <code>kubectl describe pod -n NAMESPACE POD</code>. 3. If OOM: increase memory limits (256Mi minimum for Python FastAPI apps). 4. If DB connection: verify the secret and service DNS. 5. If migration: see <code>sop-db-migration-recovery</code>.
Pod in <strong>ImagePullBackOff</strong> Image tag does not exist in Harbor, or registry credentials are wrong. 1. Check the exact image tag in the deployment: <code>kubectl get deploy -n NAMESPACE NAME -o jsonpath='{.spec.template.spec.containers[0].image}'</code>. 2. Verify the tag exists in Harbor UI. 3. If tag missing: the CI push step may have failed — check Woodpecker. 4. If creds wrong: check the <code>imagePullSecrets</code> reference and the Harbor robot account.
ArgoCD Image Updater writes a <strong>ghost override</strong> (.argocd-source file) Image Updater detected a newer tag and wrote a parameter override that conflicts with the manifest. See <code>concept-argocd-ghost-override</code> for full details. 1. Check for <code>.argocd-source-*</code> files in the app repo. 2. If present and wrong, delete the file and push. 3. Verify the Image Updater annotation on the ArgoCD Application matches the intended image policy. 4. Force ArgoCD refresh after cleanup.
Deployment succeeds but app returns <strong>502/503</strong> Pod is running but not ready. Health check failing, or service selector mismatch. 1. Check readiness probe: <code>kubectl describe pod -n NAMESPACE POD</code> (look for readiness probe failures in events). 2. Check service selector matches pod labels. 3. Check the app's health endpoint directly: <code>kubectl exec -n NAMESPACE POD -- curl localhost:PORT/health</code>.
Changes pushed but nothing deploys ArgoCD is not watching the right branch, path, or the Application resource does not exist yet. 1. Verify ArgoCD Application exists: <code>kubectl -n argocd get application</code>. 2. Check <code>.spec.source.path</code> and <code>.spec.source.targetRevision</code>. 3. ArgoCD reads from the app repo <code>k8s/</code> dir — changes to other directories do not trigger sync.

Decision Tree

When a deployment fails:
  • Identify the failure layer: Is it ArgoCD sync? Pod startup? Image pull? Network/routing?
  • If ArgoCD sync: Check Application status and manifest validity. Fix YAML and push.
  • If pod crash: Get logs with --previous flag. Check for OOM (256Mi minimum for Python), missing env vars, or DB issues.
  • If image pull: Verify the tag exists in Harbor. If not, check if CI push succeeded.
  • If ghost override: Delete .argocd-source-* file from repo and force refresh.
  • If routing/502: Check readiness probes and service selectors.
  • Do NOT manually kubectl apply in ArgoCD-managed namespaces — selfHeal will revert your changes. All fixes must go through the app repo.
  • If still failing after diagnosis: Escalate to Betty Sue with: namespace, pod name, kubectl describe output, and logs.

Escalation Criteria

Escalate immediately (do NOT self-correct) when:
  • Node is NotReady or unreachable (platform-level issue)
  • PersistentVolumeClaim is stuck in Pending (storage provisioner issue)
  • CNPG Cluster is not healthy (database infrastructure issue)
  • Harbor registry is unreachable (platform-level issue)
  • You need to modify ArgoCD Application resources (platform-level, managed by Terraform)
  • The failure involves secrets you cannot read or modify
  • You are tempted to experiment on live infrastructure — STOP and plan instead
  • deployment-lessons — memory limits, hard shutdown survival, Postgres PVC reinit
  • concept-argocd-ghost-override — full explanation of the ghost override problem
  • sop-ci-pipeline-recovery — when the failure is in CI, not deployment
  • sop-postgres-restore — when the DB needs recovery
  • sop-db-migration-recovery — when a migration failure caused the crash
  • service-onboarding-sop — correct k8s manifest structure for new services