pal-e-docs

SOP: CI Pipeline Recovery

sop-ci-pipeline-recovery Sop

sop active
Project

SOP: CI Pipeline Recovery

Purpose: Teach agents how to self-diagnose and recover from Woodpecker CI pipeline failures without escalating. Covers test failures, build failures, push failures, and smoke test failures.
Traceability: plan-pal-e-agency → Phase 5 (Error Recovery SOPs)

Failure Modes

Symptom Likely Cause Recovery Steps
Pipeline status: <strong>failure</strong>, but test and build steps show SUCCESS Smoke test failure only. Pipeline overall status reflects the worst step. 1. Check step-level status, not pipeline-level. 2. If test+build+push all passed, the image is live. 3. Investigate the smoke test step independently — it may be a transient network issue or a stale health check URL.
Test step FAILURE with ruff lint errors Ruff violations — the #1 CI failure cause platform-wide. 1. Run <code>ruff check .</code> locally. 2. Run <code>ruff format .</code> to auto-fix formatting. 3. Commit the fix. The PreToolUse hook should catch this before commit — if it did not fire, verify <code>~/.claude/hooks/</code> hardlinks are current.
Test step FAILURE with actual test errors Code bug or stale test (e.g., test expects old behavior after refactor). 1. Read the failing test name from pipeline status. 2. Run the test locally: <code>pytest tests/path/to/test.py -v</code>. 3. Fix the code or update the stale test. 4. Push and re-trigger.
Build step FAILURE (kaniko/docker build) Dockerfile syntax error, missing dependency, or base image pull failure. 1. Try <code>docker build .</code> locally. 2. Check if base image registry is accessible. 3. Check Dockerfile for recently added dependencies that are not in requirements.txt or package.json.
Push step FAILURE (Harbor registry) Harbor auth expired, project does not exist, or disk full. 1. Verify Harbor project exists: <code>harbor.tail5b443a.ts.net</code> UI. 2. Check Woodpecker secrets: <code>REGISTRY_USER</code>, <code>REGISTRY_PASSWORD</code>. 3. If disk full, escalate to Betty Sue.
Pipeline logs are empty / unreadable Known Woodpecker bug (#4409) — K8s backend log streaming fails. Logs ARE stored in SQLite DB. Workaround: <code>sudo sqlite3 /var/lib/rancher/k3s/storage/pvc-05aa5963-1864-4862-9798-ef7979949080_woodpecker_data-woodpecker-server-0/woodpecker.sqlite "SELECT data FROM log_entries WHERE step_id = &lt;ID&gt; ORDER BY line;"</code>. Alternatively, run tests locally to reproduce.
Pipeline never triggers Repo not activated in Woodpecker, or <code>.woodpecker.yaml</code> missing/malformed. 1. Check Woodpecker UI for repo activation. 2. Validate <code>.woodpecker.yaml</code> syntax. 3. Ensure push was to the correct branch (pipelines trigger on push to main by default).
tofu plan fails on PR Provider error, state drift, or missing variable. 1. Check plan output in PR comment. 2. Run <code>tofu plan</code> locally to reproduce. 3. Fix provider config or run <code>tofu init -upgrade</code>. 4. If state drift, may need <code>tofu import</code> or <code>tofu state rm</code> (L0 — requires Lucas).
tofu apply fails after merge Resource conflict, quota exceeded, or state lock. 1. Check Woodpecker pipeline logs. 2. Do NOT retry manually — CI will retry on next merge. 3. If state lock, wait 5 min for lock timeout. 4. If state corruption, follow break-glass procedure in <code>convention-apply-before-merge</code>. 5. Create TODO note for root cause.
tofu apply succeeds but resource unhealthy Config correct but resource fails to start (OOM, bad image, dependency missing). 1. Check pod/resource status via kubectl. 2. The apply was correct — the issue is the resource config. 3. Fix in a new PR, CI will re-apply on merge.
State lock contention Two applies running simultaneously (should not happen with CI serialization). 1. If CI-only: Woodpecker serializes — this shouldn't happen. Check for manual apply. 2. If manual apply conflicted: follow break-glass procedure in <code>convention-apply-before-merge</code>. 3. Run <code>tofu force-unlock &lt;ID&gt;</code> only as last resort (L0).

Decision Tree

When a CI pipeline fails:
  • Check step-level status — use mcp__woodpecker__get_pipeline_status or the Woodpecker UI. Identify WHICH step failed.
  • If test failure: Run ruff check . first. If ruff clean, run failing tests locally. Fix and push.
  • If build failure: Run docker build . locally. Fix Dockerfile or dependencies.
  • If push failure: Verify Harbor project and Woodpecker secrets. If secrets are wrong, escalate.
  • If smoke test only: Verify the image was pushed successfully (check Harbor). If image exists, the deploy is fine — investigate smoke test independently.
  • If logs are empty: Use the SQLite workaround or reproduce locally.
  • If still failing after 2 attempts: Escalate to Betty Sue with: pipeline number, step that failed, local reproduction results, and any error messages.

Manual Build+Push (CI Bypass)

When CI is fundamentally blocked and you need to deploy:
Then update k8s/deployment.yaml with the new tag and push. Force ArgoCD refresh:
WARNING: This bypasses all CI checks. Use only when CI itself is broken, not when your code is broken.

State Lock Recovery (Terraform/OpenTofu)

When tofu apply fails with "the state is already locked", the state backend (Kubernetes secret in tofu-state namespace) has a lock held by a previous operation that crashed or timed out. This blocks ALL subsequent applies, including CI.
When safe to force-unlock:
  • The locking pipeline/process has clearly crashed or been cancelled (check Woodpecker pipeline status)
  • No other tofu apply is currently running (verify in Woodpecker UI and with ps aux | grep tofu on host)
  • The lock has been held for more than 10 minutes with no active process
When NOT safe to force-unlock:
  • Another tofu apply is actively running (concurrent unlock = state corruption)
  • You are unsure whether a process is still running (verify first)
  • The lock was created by a manual apply you didn't initiate (ask the operator)
Manual unlock procedure:
Incident reference: Pipeline #80 (2026-03-17) — crashed tofu apply left stale lock, blocking all deployments on main for ~2 hours until manual force-unlock. This led to CI lock recovery automation in Phase 17b.1.

Escalation Criteria

Escalate immediately (do NOT self-correct) when:
  • Harbor registry is unreachable or returning 500 errors
  • Woodpecker secrets appear to be missing or invalid (you cannot fix secrets)
  • Woodpecker server itself is down (no pipeline status available)
  • Disk full on the node (requires platform-level intervention)
  • The same pipeline has failed 3+ times with different errors (indicates systemic issue)

Helm Rollback DB Corruption (Woodpecker)

When a Helm rollback or upgrade is performed on Woodpecker while pipelines are actively running, the server pod restarts mid-execution. This creates orphaned workflow and step records in the Woodpecker SQLite database — rows with status='running' that will never complete. The server crash-loops on startup trying to reconcile these phantom records, agents disconnect and cannot reconnect, and subsequent pipeline triggers produce false error reports referencing stale workflows.
Symptoms:
  • Phantom pipeline failures — pipelines report failure but no step actually ran
  • sql: no rows in result set errors in Woodpecker server logs
  • Agent cannot connect to server — repeated connection refused or timeout errors in agent pod logs
  • Woodpecker server pod is in CrashLoopBackOff or restarts repeatedly
  • Pipeline queue appears stuck — new pushes trigger pipelines that never start executing
Root cause: Helm rollback during an active pipeline kills the Woodpecker server pod. Workflows and steps that were in-flight are left with status='running' in the SQLite database. On restart, the server attempts to reconcile these orphaned records, fails to find matching agent state, and enters a crash-loop. Agents lose their gRPC connection and cannot re-register until the server stabilizes. Any pipelines triggered during this window produce stale or phantom error reports.
Recovery procedure:
Prevention:
  • Always drain the pipeline queue before performing Helm operations on Woodpecker — wait for all running pipelines to complete or cancel them explicitly via mcp__woodpecker__cancel_pipeline
  • Check queue status before Helm upgrades: mcp__woodpecker__get_queue_status — if any items are queued or running, wait
  • If a rollback is unavoidable during active pipelines, expect DB corruption and plan for the recovery procedure above immediately after the rollback completes
Incident reference: Issue #242 (2026-03-28) — Helm rollback on Woodpecker during active CI pipeline created orphaned DB records. Server entered crash-loop, agent disconnected, and phantom failures propagated to multiple repos until orphaned workflows were manually vacuumed from SQLite.
  • deployment-lessons — Woodpecker variable syntax, repo activation
  • service-onboarding-sop — initial Woodpecker setup for new services
  • sop-ci-pipeline-recovery — this note
  • pr-lifecycle — Stage 3 (PR Submission) triggers CI