pal-e-docs

Phase: Observability — Alerting + Deployment Protection

phase-observability-3-alerting Phase

Project
Goal: Critical platform alerts fire and reach a human. Deployments fail safely. First DORA metric moved: MTTR.
Owner: Dev agent
Repo: forgejo_admin/pal-e-platform
Depends on: phase-observability-2-verify-baseline (COMPLETED)

Why

DORA impact: This phase directly improves MTTR (alerts reduce detection time from "someone notices" to immediate) and Change Failure Rate (deployment protection prevents bad rollouts from propagating). Without alerting, the DORA exporter collects the scorecard but the platform can't react to the scores. The pal-e-docs Alembic crash (2026-02-26) went undetected — proof that detection is the gap.

Scope

1. PrometheusRules in Terraform. Four critical platform alerts defined via additionalPrometheusRules in kube-prometheus-stack Helm values:
  • Pod restart storm: increase(kube_pod_container_status_restarts_total[15m]) > 3
  • OOMKilled: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} > 0
  • Disk pressure: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 15
  • Target down: up == 0 for 5m
2. Alertmanager routing. Configure receiver in kube-prometheus-stack Helm values. Slack webhook URL as optional sensitive TF variable — available but unused. The actual push notification channel is Telegram (see subphase phase-observability-3a-telegram-alerting). Alertmanager UI via Tailscale funnel provides guaranteed browser-based visibility regardless of push config.
3. Alertmanager Tailscale funnel. Expose Alertmanager UI at alertmanager.{tailscale_domain} — guaranteed browser-based visibility regardless of Slack config. Follows existing funnel pattern (Grafana, Forgejo, etc.).
Forgejo Issue: forgejo_admin/pal-e-platform #33
5. Verification. Intentionally trigger an alert (e.g., scale a test deployment to cause restarts), verify it fires in Prometheus /rules, appears in Alertmanager UI, and routes to Slack (if configured).

Deliverables

  • PR #35 merged (2026-03-14) — Closes #33
  • 4 PrometheusRules: PodRestartStorm, OOMKilled, DiskPressure, TargetDown
  • Alertmanager config: default null receiver + conditional Slack receiver (unused — Telegram in subphase 3a)
  • Alertmanager Tailscale funnel at alertmanager.{tailscale_domain}
  • slack_webhook_url sensitive variable (default empty)
  • Remaining: Subphase 3a (Telegram alerting) still not-started. Phase stays in-progress.
  • plan-pal-e-platform — parent plan (Platform Hardening)
  • phase-observability-2-verify-baseline — predecessor (confirmed baseline is healthy)
  • phase-observability-4-dashboard — next phase (golden signals dashboard for pal-e-docs)
  • service-onboarding-sop — to be updated with alerting and deployment protection requirements