pal-e-docs

Concept: ArgoCD Ghost Override

concept-argocd-ghost-override Doc

Project

What Is a Ghost Override?

ArgoCD Image Updater is a companion controller that watches container registries for new image tags and automatically updates running deployments. When it finds a new tag, it writes an override file — .argocd-source-<app-name>.yaml — into the application's source directory. This file tells ArgoCD to use a different image tag than what's in the checked-in deployment.yaml.
The ghost override happens when:
  • Image Updater writes .argocd-source-pal-e-docs.yaml to override the image tag
  • Image Updater is later disabled or removed (annotations stripped from the ArgoCD Application)
  • But the override file persists — it was written to the repo checkout, not managed by the controller's lifecycle
  • ArgoCD continues applying the stale override on every sync, silently ignoring the image tag in deployment.yaml
The result: you update deployment.yaml with a new image tag, ArgoCD says "Synced", but the pod runs a completely different image. The override is invisible unless you know to look for .argocd-source-* files.

Why It's Dangerous

  • Silent divergence: Git says one image, cluster runs another. GitOps promise is broken.
  • Debugging nightmare: kubectl get deploy -o yaml shows the overridden image, not what's in Git. The file causing it isn't even in Git — it's in ArgoCD's local repo clone.
  • Survives annotation removal: Removing Image Updater annotations from the Application doesn't clean up existing override files.

How We Fixed It (PR #91)

  • Removed Image Updater annotations from the ArgoCD Application (done previously)
  • Added .argocd-source-* to .gitignore to prevent future write-backs from landing in Git
  • ArgoCD picked up the gitignore change, stopped reading the override file

Prevention

  • Always add k8s/.argocd-source-* to .gitignore in any repo using ArgoCD
  • When disabling Image Updater, manually verify no stale override files remain in ArgoCD's repo cache
  • Pin image tags explicitly in deployment manifests — don't rely on :latest or dynamic updaters until the full pipeline (registry auth, write-back) is proven

See Also

  • bug-argocd-image-updater-ghost-override — the original bug report
  • bug-image-updater-harbor-auth — why Image Updater was broken in the first place
  • incident-phase5-deployment-outage-2026-03-06 — the outage caused when the override was removed