pal-e-docs

Phase: Observability — Telegram Alerting

phase-observability-3a-telegram-alerting Phase

Project
Goal: Platform alerts reach Lucas's phone via Telegram push notifications. Primary notification channel for MTTR.
Owner: Dev agent
Repo: forgejo_admin/pal-e-platform
Depends on: phase-observability-3-alerting (PrometheusRules + Alertmanager funnel must exist first)

Why

Phase 3 deploys PrometheusRules and the Alertmanager UI funnel, with Slack as an available-but-unused receiver. Telegram is the chosen push channel because: native Alertmanager support (telegram_configs), zero infrastructure overhead, reliable mobile push notifications, 5-minute setup. Slack stays in the config as a dormant option — no workspace needed.

Scope

1. Manual pre-req (Lucas): Create a Telegram bot via @BotFather, get the bot token. Send a message to the bot, retrieve the chat ID. Provide both values for TF variables.
2. Terraform changes:
  • Add telegram_bot_token (sensitive) and telegram_chat_id variables to variables.tf
  • Add telegram_configs receiver to Alertmanager config in kube-prometheus-stack Helm values
  • Set Telegram as the default route receiver (Slack remains as an unused secondary)
  • Configure parse_mode: HTML for rich alert formatting
3. Alertmanager routing design:
  • route.receiver: telegram (default — all alerts go here)
  • group_wait: 30s, group_interval: 5m, repeat_interval: 4h
  • Slack receiver stays defined but no route points to it (available for future use)
4. Verification: Trigger a test alert, confirm Telegram push notification arrives on phone.

Deliverables

  • PR #43 merged (2026-03-14) — Closes #42
  • Telegram receiver added as default Alertmanager route (route.receiver: telegram)
  • telegram_configs with send_resolved: true, parse_mode: HTML
  • telegram_bot_token and telegram_chat_id injected via static set_sensitive blocks (not in yamlencode)
  • Slack receiver bumped to index 2, remains available but dormant
  • Alerting target: Telegram group chat (enterprise pattern — onboard developers by adding to group, no infra changes)
  • QA nit: no validation guard if chat_id=0 when token is set — non-blocking
  • phase-observability-3-alerting — parent phase
  • plan-pal-e-platform — grandparent plan (Platform Hardening)