pal-e-docs

Phase 14b: Observability Cleanup — Alert Noise Reduction

phase-pal-e-platform-14b-observability-cleanup Phase

phase
Project
Goal: Reduce active alerts from 24 to <5 by removing broken deployments and fixing Blackbox Exporter probe URLs.
Owner: Betty Sue + Dev agent
Repo: forgejo_admin/pal-e-platform
Depends on: phase-pal-e-platform-14-synthetic-monitoring

Problem

24 active alerts = alert fatigue. 16 from known-broken services (3 MCP remotes with no images, 1 unused dev namespace), 5 from misconfigured probe URLs, 2 historical OOMKilled, 1 intentional Watchdog. When everything screams, nothing gets attention — a DORA MTTR antipattern.

Fix

  • 14b-1: Delete MCP remote deployments, services, ServiceMonitors, Ingresses, and ArgoCD apps (gmail-mcp-remote, linkedin-scheduler-remote, notion-mcp-remote). Namespaces preserved for future re-deploy. (-16 alerts)
  • 14b-2: Delete basketball-api-dev deployment, postgres, services, ServiceMonitor, ArgoCD app. (-4 alerts)
  • 14b-3: Fix 3 Blackbox probe URLs — keycloak (DNS collision with Tailscale MagicDNS → use external funnel URL), basketball-api (root 404 → internal /docs), platform-validation (TLS timeout → internal :80). PR #70, Closes #69. (-5 alerts)
  • 14b-4: Clean stale completed/failed pods (ollama-test3, test-clone2, palworld-client).

Deliverables

  • Alerts reduced 24 → 8 (immediate), → 3 after tofu apply (Watchdog + 2 historical OOMKilled)
  • PR #70 merged — 3 probe URL fixes
  • All ImagePullBackOff, Pending, Failed pods eliminated
  • 4 ArgoCD apps removed (3 MCP + basketball-api-dev)
  • phase-pal-e-platform-14-synthetic-monitoring — parent phase
  • plan-pal-e-platform — parent plan
  • PR #70 — probe URL fixes
  • Issue #69 — closed