pal-e-docs

Phase: Incident Management SOP

phase-pal-e-platform-incident-mgmt Phase

phase
Project

Phase: Incident Management SOP

Status: COMPLETED 2026-03-14.

Lineage

plan-pal-e-platform → Phase 12

Deliverables

  • DONE: Created sop-incident-response — full incident response SOP
  • DONE: Defined severity levels: P1 (service down), P2 (degraded), P3 (cosmetic)
  • DONE: Documented detection sources: Prometheus alerts, Blackbox Exporter, Grafana, DORA exporter, Woodpecker, manual
  • DONE: Defined 6-step incident flow: Detection → Triage → Diagnosis → Remediation → Verification → Postmortem
  • DONE: Created 6 runbooks: Pod CrashLoopBackOff, CNPG Failover, Woodpecker Pipeline Stuck, ArgoCD Sync Failure, Tailscale Funnel Unreachable, Disk Pressure
  • DONE: Documented all alerting rules with severity and runbook links
  • DONE: Linked to related SOPs (postgres-restore, db-migration-recovery, ci-pipeline-recovery, deploy-recovery, mcp-server-recovery)
  • sop-incident-response — the SOP created by this phase
  • dora-framework — MTTR measurement depends on structured incident response