Alert State Report — 2026-05-01

Snapshot

• 9 firing, 0 pending, 0 silenced
• 123 alerting rules total across 36 rule groups (7.3% active)
• 3 westside-relevant, 5 platform-infra, 1 heartbeat

Why some "expected" failure modes are missing

Currently firing — full list

Westside-relevant (3)

Not westside — platform infra (5)

Heartbeat (1)

Where each rule comes from (123 total alerting rules)

What's NOT covered by any rule

• westside-contracts, westside-email, westside-ai-assistant — no probes, no metrics scrapes, no rules. Invisible.
• Pods in ImagePullBackOff / Init — disabled kubernetesApps rule family removed KubeContainerWaiting.
• Deployment replica mismatches — same reason (KubeDeploymentReplicasMismatch disabled).
• HTTP 5xx rates on basketball-api endpoints — prometheus-fastapi-instrumentator not wired yet (separate ticket).
• Webhook processing errors per event type — WebhookErrorRate doesn't differentiate signature-fail vs handler-throw vs idempotency-skip.

Routing & inhibition (Alertmanager config)

• Single receiver: telegram (everything goes there)
• One inhibit rule (added on this branch): severity=critical suppresses severity=warning when alertname and namespace match. Works for OOMKilled (no warning version exists), but misses GmailOAuthToken* because the two have different alertnames — that's why both are firing.
• No grouping by severity — critical and warning hit telegram with the same routing.

Three things stand out

• WebhookStale lit up ~30 min ago. Worth checking whether it caught a real outage or whether the time-of-day filter (hour() >= 16 or hour() < 4 UTC) is wrong for actual MST business hours — it's currently early morning MST on a Friday, which means it just rolled into "business hours" by the rule's definition. If no real checkouts happen until later, this fires daily at the same time. May need an "after first checkout of the day" variant.
• Gmail OAuth has been broken for 2+ days. Westside email is dead. The auto-reauth cron isn't doing its job. Real fire.
• The Mac alerts are 16-34 days old. Three firing alerts for one offline laptop. Either silence the Mac during expected-offline windows or cut the duplicate TargetDown rules.

Cleanup proposal (from prior discussion)

• Ship branch 290-payment-pipeline-observability — already silences kubeStateMetrics/kubernetesApps, raises blackbox for from 2m→5m, drops noisy probes, adds basketball-api-golden-signals dashboard, adds WebhookErrorRate + WebhookStale.
• Fix the Gmail inhibit rule. Restructure so Expired/ExpiringSoon share an alertname with different severity, OR equal-on secret label.
• Triage non-westside criticals as separate tickets: pal-e-docs OOM (bump memory or fix leak), argocd OOM (raise limits), MacAgentDown (time-window the alert).
• Add a westside-unified dashboard + add probes for westside-contracts, westside-email, westside-ai-assistant.
• Fix the actual Gmail OAuth lifecycle. Auto-reauth (#222) shipped but token is 52 days old — cron either isn't running or is failing silently.
• Re-add coverage for failed pod states. The disabled kubernetesApps family included real signal (KubeContainerWaiting, KubePodNotReady). Either selectively re-enable a subset or write a tighter custom rule scoped to known-good namespaces.