pal-e-docs

Incident: Woodpecker webhook signatures invalid — merge=deploy broken

incident-2026-03-14-woodpecker-webhook-signatures Doc

platform incident ci
Project

Incident: Woodpecker webhook signatures invalid

Severity: P2 — Degraded
Status: RESOLVED 2026-03-14 21:50 UTC
Detected: 2026-03-14 ~21:27 UTC
Resolved: 2026-03-14 ~21:50 UTC (23 minutes)
Duration of outage: Unknown — likely broken since initial Woodpecker deployment (~3 weeks), silently worsening through every pod restart

Timeline

Time Event
~Feb 24 Woodpecker initially deployed. WOODPECKER_AGENT_SECRET not set.
Multiple Pod restarts over 3 weeks. Each generates new random signing key, silently invalidating tokens.
Mar 14 21:27 PR #67 merged. Pipeline does NOT trigger. Discovered during post-merge verification.
Mar 14 21:35 Root cause identified: missing WOODPECKER_AGENT_SECRET.
Mar 14 21:42 Fix applied: persistent WOODPECKER_AGENT_SECRET set via tofu apply. Woodpecker restarted.
Mar 14 21:45 56 stale webhooks deleted across 28 repos.
Mar 14 21:48 28 repos deactivated/re-activated with fresh webhooks.
Mar 14 21:50 PR #68 pushed. Pipeline #18 triggers automatically. <strong>INCIDENT RESOLVED.</strong>

Root Cause

WOODPECKER_AGENT_SECRET was not set in the Helm deployment. Woodpecker generates a random JWT signing key at every startup when this is not set. Every pod restart invalidated all existing tokens.

Fix Applied

  • Added persistent WOODPECKER_AGENT_SECRET (PR #68)
  • Regenerated API token, updated 3 config files
  • Deleted 56 stale webhooks, re-activated 28 repos
  • Verified: Pipeline #18 triggered automatically from PR push

Lessons Learned

  • Always set explicit signing keys for stateful services — random-at-startup secrets are a ticking time bomb in Kubernetes
  • Need a "CI trigger health" alert — detect "merge happened but no pipeline created within 2 minutes"
  • Deactivate/re-activate creates duplicate webhooks in Forgejo — must delete old ones first
  • phase-pal-e-platform-14a-webhook-fix — the fix phase (COMPLETED)
  • PR #68 — the code change
  • sop-incident-response — first incident processed through this SOP