Incident: Phase 5 Deployment Outage (2026-03-06)

incident-phase5-deployment-outage-2026-03-06 Doc

incident

Project

Summary

pal-e-docs was down for ~15 minutes on 2026-03-06. The pod entered ImagePullBackOff because the image tag in deployment.yaml referenced a commit SHA that never existed as a Harbor image tag. The Recreate deployment strategy killed the running pod before the new one could start.

Timeline

Time	Event
Prior sessions	PR #84 (tsvector search), PR #86 (ruff format), PR #88 (pin image tag), PR #91 (ghost override fix) all merged. CI built images for each merge commit.
PR #88 merge	deployment.yaml pinned to <code>c85a39da...</code> — the <strong>squash commit SHA</strong> from PR #86's branch, NOT the merge commit SHA on main (<code>c757d179...</code>). This tag never existed in Harbor.
Before this session	Pod was still running image <code>e0654197...</code> (PR #77) due to the ArgoCD ghost override. The wrong tag in deployment.yaml was masked.
PR #91 merge	Removed ghost override mechanism. ArgoCD now reads the actual deployment.yaml tag.
~T+0	ArgoCD syncs, triggers Recreate rollout. Old pod killed. New pod fails to pull <code>c85a39da...</code> — image not found in Harbor.
~T+5 min	Outage detected during next session. Pod in ImagePullBackOff.
~T+10 min	Emergency rollback: disabled ArgoCD auto-sync, patched to cached <code>e0654197...</code> (IfNotPresent). Service restored.
~T+15 min	Updated deployment to <code>2eddd766...</code> (latest merge commit, includes search code). Search endpoint verified live.

Root Causes

Wrong SHA in PR #88: Image tag was set to the squash/branch commit SHA (c85a39da) instead of the merge commit SHA (c757d179). Woodpecker CI tags images with ${CI_COMMIT_SHA}, which is the merge commit on main — not the branch commit.
Ghost override masked the bug: .argocd-source-pal-e-docs.yaml was overriding the image tag to e0654197 (PR #77). The invalid tag in deployment.yaml was never used until PR #91 removed the override.
Recreate strategy has no safety net: strategy: Recreate kills the old pod before the new one is ready. With RollingUpdate + readiness probes, the old pod would have continued serving.

Contributing Factors

Woodpecker MCP logs broken (woodpecker-mcp #3) — couldn't inspect CI output to verify image tags.
Harbor auth confusion: Empty-password admin showed only 2 projects, hiding the fact that pal-e-docs images existed. Red herring investigation.
SHA confusion: Forgejo squash merges produce two SHAs — the squash commit (branch side) and the merge commit (main side). CI uses the merge commit. PR #88 used the wrong one.

Resolution

Disabled ArgoCD auto-sync temporarily
Patched deployment to e0654197 (cached, working) with imagePullPolicy: IfNotPresent
Upgraded to 2eddd766... (latest merge commit, includes tsvector search)
Verified search endpoint returns results
Updated deployment.yaml in Git to match, re-enabled ArgoCD auto-sync

Action Items

Immediate: Update k8s/deployment.yaml image tag in Git, re-enable ArgoCD auto-sync
TODO: Change deployment strategy from Recreate to RollingUpdate with readiness probes
TODO: Add CI step or hook that validates image tag = merge commit SHA
TODO: Fix Woodpecker MCP logs (woodpecker-mcp #3)

Lessons Learned

Squash commit SHA ≠ merge commit SHA. Woodpecker ${CI_COMMIT_SHA} is the merge commit. Never pin to a branch/squash SHA.
Ghost overrides mask broken configs. Fixing one bug (override) can expose another (wrong tag). Always verify the full deployment path end-to-end after removing workarounds.
Recreate strategy is dangerous without image pre-pull. Use RollingUpdate for zero-downtime deployments.
Harbor admin auth requires actual password. Empty password shows limited results.