pal-e-docs

SOP: Incident Response

sop-incident-response Sop

sop active platform
Project

SOP: Incident Response

Status: Active. Created 2026-03-14 as part of Phase 12 (Incident Management).
Purpose: Structured incident response reduces MTTR. This SOP defines the detection → triage → diagnosis → remediation → postmortem pipeline. Every incident follows this flow.

Severity Levels

Severity Definition Examples Response Time Notification
<strong>P1 — Service Down</strong> User-facing service completely unavailable pal-e-docs 502, Forgejo unreachable, Keycloak down, ArgoCD sync loop Immediate Telegram alert (automatic)
<strong>P2 — Degraded</strong> Service functional but impaired (slow, partial failures, elevated errors) Response time &gt;5s, pod restarts, CNPG failover, CI pipeline stuck &lt;1 hour Telegram alert (automatic)
<strong>P3 — Cosmetic / Non-blocking</strong> Visible issue that doesn't affect functionality Dashboard rendering glitch, stale metric, log noise Next session None (discovered during work)

Detection Sources

Source What It Detects Alert Channel Dashboard
<strong>Prometheus Alertmanager</strong> Pod restarts, OOMKilled, disk pressure, target down Telegram + Slack <a href="https://alertmanager.tail5b443a.ts.net">Alertmanager UI</a>
<strong>Blackbox Exporter</strong> Endpoint unreachable (probe_success == 0) Telegram (via PrometheusRule) Service Uptime dashboard
<strong>Grafana dashboards</strong> Latency spikes, error rate increase, saturation Visual (manual check) <a href="https://grafana.tail5b443a.ts.net">Grafana</a>
<strong>DORA exporter</strong> Deployment failures, pipeline stuck DORA dashboard anomaly DORA Metrics dashboard
<strong>Woodpecker CI</strong> Pipeline failures Woodpecker UI notification <a href="https://woodpecker.tail5b443a.ts.net">Woodpecker UI</a>
<strong>Manual discovery</strong> User reports, session observation Lucas notices N/A

Incident Response Flow

Step 1: Detection

Incident is detected via one of the sources above. Automatic alerts (Telegram) provide the alert name, namespace, and severity. If detected manually, note the time and symptoms.

Step 2: Triage (1-2 minutes)

Determine severity level (P1/P2/P3) and scope:
  • Is a user-facing service down? → P1
  • Is a service degraded but functional? → P2
  • Is it cosmetic or non-blocking? → P3
  • What is the blast radius? Single service, namespace, or cluster-wide?
  • Is this a known failure mode? Check runbooks below.

Step 3: Diagnosis (5-15 minutes)

Use the diagnostic toolkit to identify root cause:
  • kubectl get pods -n <namespace> — check pod status
  • kubectl describe pod <pod> -n <namespace> — events, conditions
  • kubectl logs <pod> -n <namespace> --tail=100 — recent logs
  • kubectl get events -n <namespace> --sort-by=.lastTimestamp — cluster events
  • Grafana → service dashboard → check traffic/latency/errors/saturation
  • Loki → {namespace="<ns>"} → search for error patterns
  • Prometheus → direct PromQL queries for specific metrics

Step 4: Remediation

Apply the appropriate fix based on root cause. Common remediation actions:
  • Pod crash: Check logs, fix code/config, redeploy
  • Resource exhaustion: Increase limits, optimize, or scale
  • Bad deploy: Rollback via ArgoCD (sync to previous commit) or revert PR
  • Infrastructure: tofu apply to reconcile state, or manual kubectl fix
  • Database: Follow sop-postgres-restore if data loss, or sop-db-migration-recovery if migration failure
  • Board tracking: Create a board item on the affected project's board representing the fix action (not the incident itself). Use create_board_item(board_slug="board-PROJECT", item_type="todo", title="Fix: [description of remediation]", column="in_progress", labels="type:incident"). Incidents start in in_progress because they're already being worked when discovered. Move to done after Step 5 verification passes.

Step 5: Verification

Confirm the fix:
  • Service responding (check dashboard or curl)
  • Alert resolved in Alertmanager (auto-resolves when condition clears)
  • No new errors in logs
  • Metrics returning to baseline

Step 6: Postmortem (P1 and P2 only)

Create an incident note in pal-e-docs:
  • create_note(title="Incident: ...", slug="incident-YYYY-MM-DD-description", note_type="incident", tags="incident", project="pal-e-platform")
  • Include: timeline, root cause, remediation, lessons learned, action items
  • Link to relevant Grafana dashboard screenshots
  • Update this SOP if a new failure mode was discovered

Common Failure Runbooks

Pod CrashLoopBackOff

Step Command / Action
1. Check logs <code>kubectl logs &lt;pod&gt; -n &lt;ns&gt; --previous</code> (previous = crashed container)
2. Check events <code>kubectl describe pod &lt;pod&gt; -n &lt;ns&gt;</code> → look at Events section
3. Common causes Bad migration (Alembic), missing secret, OOMKilled, config error
4. Fix Fix code/config → push → CI builds → ArgoCD syncs. Or rollback: <code>argocd app rollback &lt;app&gt;</code>
5. Dashboard Grafana → golden signals dashboard for the service

CNPG Database Failover

Step Command / Action
1. Check cluster status <code>kubectl get cluster -n &lt;ns&gt;</code> — check READY and STATUS
2. Check pods <code>kubectl get pods -n &lt;ns&gt; -l cnpg.io/cluster=&lt;name&gt;</code>
3. Check timeline <code>kubectl cnpg status &lt;cluster&gt; -n &lt;ns&gt;</code> (if cnpg plugin installed)
4. If data loss Follow <code>sop-postgres-restore</code> — restore from MinIO backup
5. Dashboard Grafana → CNPG dashboard (PodMonitor metrics)

Woodpecker Pipeline Stuck

Step Command / Action
1. Check queue Woodpecker MCP: <code>get_queue_status</code>
2. Check agent <code>kubectl get pods -n woodpecker -l app=woodpecker-agent</code>
3. Agent PVC stale? Delete agent PVC + restart: <code>kubectl delete pvc agent-config -n woodpecker</code> then <code>kubectl rollout restart deployment woodpecker-agent -n woodpecker</code>
4. Cancel stuck pipeline Woodpecker MCP: <code>cancel_pipeline(repo_id, pipeline_number)</code>
5. Dashboard <a href="https://woodpecker.tail5b443a.ts.net">Woodpecker UI</a> → check pipeline logs

ArgoCD Sync Failure

Step Command / Action
1. Check app status <code>kubectl get application -n argocd</code> — look for Degraded/OutOfSync
2. Check sync details <code>argocd app get &lt;app&gt;</code> or ArgoCD UI
3. Common causes Invalid manifests, missing namespace, SOPS decryption failure, image pull error
4. Fix manifests Fix in app repo → push → ArgoCD auto-syncs
5. Force sync <code>argocd app sync &lt;app&gt; --force</code> (last resort)
6. Dashboard <a href="https://argocd.tail5b443a.ts.net">ArgoCD UI</a>

Tailscale Funnel Unreachable

Step Command / Action
1. Check Tailscale operator <code>kubectl get pods -n tailscale</code>
2. Check ingress <code>kubectl get ingress -n &lt;ns&gt;</code> — verify ingress exists and has correct class
3. Check backend service <code>kubectl get svc -n &lt;ns&gt;</code> — verify service exists and has endpoints
4. Check Tailscale proxy <code>kubectl get pods -n tailscale -l app.kubernetes.io/name=tailscale</code> — look for the proxy pod for this funnel
5. Restart proxy Delete the Tailscale proxy pod — it will be recreated by the operator

Disk Pressure (Node)

Step Command / Action
1. Check disk usage <code>df -h</code> on the node
2. Container images <code>crictl images</code> — prune unused: <code>crictl rmi --prune</code>
3. PVC usage Check large PVCs: <code>kubectl get pv --sort-by=.spec.capacity.storage</code>
4. Log rotation Check <code>/var/log</code> and Loki/Promtail storage
5. Dashboard Grafana → Node dashboard → disk panels

Alerting Rules Reference

Alert Condition Severity Runbook
PodRestartStorm &gt;3 restarts in 15m warning Pod CrashLoopBackOff runbook
OOMKilled Container OOMKilled critical Check resource limits, increase or optimize
DiskPressure &lt;15% free space critical Disk Pressure runbook
TargetDown Scrape target unreachable &gt;5m warning Check service + ServiceMonitor
EndpointDown probe_success == 0 for &gt;2m critical Tailscale Funnel Unreachable runbook
EndpointSlowResponse probe_duration_seconds &gt; 5s for &gt;5m warning Check service resource saturation
  • sop-postgres-restore — CNPG backup restore procedure
  • sop-db-migration-recovery — failed Alembic migration recovery
  • sop-ci-pipeline-recovery — CI pipeline failure triage
  • sop-deploy-recovery — deployment failure recovery
  • sop-mcp-server-recovery — MCP server failure recovery
  • deployment-lessons — operational lessons learned
  • dora-framework — DORA metrics (MTTR measurement)